Cannot install tensorflow-gpu requirement

Tried installing the requirements-gpu.txt and get this error:

Collecting tensorflow-gpu==1.4.0 (from -r requirements-gpu.txt (line 6)) Cache entry deserialization failed, entry ignored Could not find a version that satisfies the requirement tensorflow-gpu==1.4.0 (from -r requirements-gpu.txt (line 6)) (from versions: ) No matching distribution found for tensorflow-gpu==1.4.0 (from -r requirements-gpu.txt (line 6))

I went here to troubleshoot the issue: https://github.com/tensorflow/tensorflow/issues/8251
Installed Python 64bit. Opened new command prompt window and typed in: pip3 install --upgrade tensorflow-gpu

Successfully uninstalled setuptools-28.8.0
Successfully installed bleach-1.5.0 enum34-1.1.6 html5lib-0.9999999 markdown-2.6.11 numpy-1.13.3 protobuf-3.5.1 setuptools-38.4.0 six-1.11.0 tensorflow-gpu-1.4.0 tensorflow-tensorboard-0.4.0rc3 werkzeug-0.14.1 wheel-0.30.0

Went back to my faceswap env to enter the requirements-gpu.txt and still get the same error:
(faceswap) C:\faceswap>pip install -r requirements-gpu.txt
Collecting tensorflow-gpu==1.4.0 (from -r requirements-gpu.txt (line 6))
  Could not find a version that satisfies the requirement tensorflow-gpu==1.4.0 (from -r requirements-gpu.txt (line 6)) (from versions: )
No matching distribution found for tensorflow-gpu==1.4.0 (from -r requirements-gpu.txt (line 6))

## Other relevant information

- **Operating system and version:** Windows 10
Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1900 64 bit (AMD64)] on win32
- **Faceswap version:** 1/5/2018
- **Faceswap method:** CPU/GPU    "CPU method only works"
- ...


doesn't work on 2GB GTX 960 even with LowMem model (what params could be reduced?)

LowMem is different from the common model with 2 lines:
ENCODER_DIM = 512       # instead of 1024
#x = self.conv(1024)(x)       - commented out.

But it's still not enough to run under Ubuntu 16.04, cuda8, 1.7Gb of free video RAM.
It fails with OOM on any batch size, even with bs=1 and bs=2.

What about having some configurable params here? Like reducing filters numbers or ENCODER_DIM or smth else?
Also that would be great to have some doc which describes few main params and their influence on quality etc. For example fakeapp allows to select number of layers, nodes etc.

P.S. I managed to run it with ENCODER_DIM = 64 and bs=16, but results are not so good (after 15 hours).


Bounding boxes coordinates

It has been 2 weeks I have been working on it but cannot find the solution.

I want the bounding boxes on the original image, of the result that is produced by the "Extract" process of faceswap code.

"Extract" writes the faces extracted from the input image(s). I just want the coordinates from which this face is extracted (from original image).

If you could help me. I would be very grateful and would also help other people searching for the same problem.
Thank you.


Problem with FadeOutAndShift

t3 text is not going through FadeOutAndShift.
Also tell me how I can FadeOutAndShift t1 and t3 together

```# python -m manim try3.py test1 -pm

from manimlib.imports import *

class test1(Scene):
	def construct(self):
		t1=TextMobject("Hi!")
		t2=TextMobject("My name is")
		t3=TextMobject("Girish")

		t1.set_color(RED)
		t3.set_color(BLUE)

		self.play(Write(t1), run_time=2)
		self.play(ApplyMethod(t1.shift, 1*UP))
		self.play(FadeIn(t2))
		self.play(Transform(t2, t3), run_time=2)
		self.wait(2)
		self.play(FadeOutAndShift(t1))
                self.play(FadeOutAndShift(t3))

```


ImportError: No module named animation

I've installed manim on Win10.  After run "python  extract_scene.py -s example_scenes.py",

the next error is shown in the python interactive interpretor:

> Traceback (most recent call last):
  File "extract_scene.py", line 15, in <module>
    from scene.scene import Scene
  File "G:\python\manim\scene\scene.py", line 16, in <module>
    from animation.transform import MoveToTarget
  File "G:\python\manim\animation\transform.py", line 8, in <module>
    from animation.animation import Animation
ImportError: No module named animation

What I can do? I'm looking forward to get help to solve this problem.


How to add sound to video?


Swedish characters not working

Include at least:
1. Steps to reproduce the issue (e.g. the command you ran)
2. The unexpected behavior that occurred (e.g. error messages or screenshots)
3. The environment (e.g. operating system and version of manim)

I am new to manim and want to include swedish characters in a text, but it gives an error message when rendering.
Code:
class Swe(Scene):
	def construct(self):
		text = TextMobject(r"$\"o$")
		self.add(text)
		self.wait()

Error message:
Traceback (most recent call last):
  File "C:\Manim\manim\manim2020\manimlib\extract_scene.py", line 153, in main
    scene = SceneClass(**scene_kwargs)
  File "C:\Manim\manim\manim2020\manimlib\scene\scene.py", line 54, in __init__
    self.construct()
  File "Geony.py", line 115, in construct
    text = TextMobject(r"$\"o$")
  File "C:\Manim\manim\manim2020\manimlib\mobject\svg\tex_mobject.py", line 144, in __init__
    self, self.arg_separator.join(tex_strings), **kwargs
  File "C:\Manim\manim\manim2020\manimlib\mobject\svg\tex_mobject.py", line 45, in __init__
    self.template_tex_file_body
  File "C:\Manim\manim\manim2020\manimlib\utils\tex_file_writing.py", line 19, in tex_to_svg_file
    dvi_file = tex_to_dvi(tex_file)
  File "C:\Manim\manim\manim2020\manimlib\utils\tex_file_writing.py", line 67, in tex_to_dvi
    "See log output above or the log file: %s" % log_file)
Exception: Latex error converting to dvi. See log output above or the log file: C:\Manim\manim\manim2020\manimlib\files\Tex\a26fbd67dc90adbc.log

I am running python 3.7 (64 bit) and MikTex 2.9. All other features of manim are working fine.
Any help would be much appreciated. Also, please keep in mind that I am new to manim and programing in general.


Reconsidering TEX_USE_CTEX / using XeLaTeX

I worked on manim back in 2018. I added the function for using CTeX (XeLaTeX package for Chinese) and XeLaTeX instead of LaTeX using the flag `TEX_USE_CTEX` in constants.py (#315).

I have stopped working on manim since 2019, but over the months there are apparently more and more people who want to use LaTeX rendering in non-English languages, and even on very old issues I still occasionally see people asking how to do that... Looking back at my change I really should have **decoupled using CTeX (TeX template) from XeLaTeX (rendering tool)**. This has caused a *lot* of confusions and made weird hacks/fixes necessary for only using XeLaTeX, especially for a language that is not Chinese or English, with the most recent #858 and #840. It really should have been a flag `TEX_USE_XELATEX` and another flag `TEMPLATE_TEX_NAME`, and the flag `TEX_USE_CTEX` is such that when it is `True`, `TEX_USE_XELATEX` is `True` and `TEMPLATE_TEX_NAME` is `"ctex_template.tex"`; otherwise `TEX_USE_XELATEX` is `False` and `TEMPLATE_TEX_NAME` is `"tex_template.tex"`. Then set `TEMPLATE_TEX_FILE` to `os.path.join(os.path.dirname(os.path.realpath(__file__)), TEMPLATE_TEX_NAME)`. Corresponding logic: constants.py lines 74–79.

It might be even better to set it dynamically using a function or as a parameter of `TexMobject()`, (see issues like #891). I looked at the source code and this is definitely possible. The options I can think of are
1. Use the current `TEX_USE_CTEX`
2. Add flags `TEX_USE_XELATEX` and `TEMPLATE_TEX_NAME`, and rework `TEX_USE_CTEX`
3. Add parameters for `TexMobject()` like `use_xelatex=False` and `tex_template="tex_template.tex"`
4. Use the flags of 2. as a default, and make it possible to change the default using 3.

Not really sure if this is the right place to raise this issue.


ColorByCaracter help

I want to color only theta of ```{ e }^{ i\theta  }```

I was going through ColorByCaracter in 3_text_like_arrays.py .
But I fail to understand how you people separate the tex formula into arrays. I know about arrays but I can only copy the tex code from [Daum Equation Editor](http://s1.daumcdn.net/editor/fp/service_nc/pencil/Pencil_chromestore.html) and paste it. I don't know how to divide them into arrays.

Please help me.


OSError: No file matching Suv.svg in image directory

I've tried putting the .SVG image into */media/designs/svg_images. But when I want to quote it in the .py file it still reports errors:

```
Traceback (most recent call last):
  File "/home/jason/Documents/manim/manimlib/extract_scene.py", line 155, in main
    scene = SceneClass(**scene_kwargs)
  File "/home/jason/Documents/manim/manimlib/scene/scene.py", line 53, in __init__
    self.construct()
  File "SVGTEST.py", line 44, in construct
    height=height_size
  File "/home/jason/Documents/manim/manimlib/mobject/svg/svg_mobject.py", line 45, in __init__
    self.ensure_valid_file()
  File "/home/jason/Documents/manim/manimlib/mobject/svg/svg_mobject.py", line 63, in ensure_valid_file
    self.file_name)
OSError: No file matching MYSVG.svg in image directory

```
(Manjaro Linux, Texlive)


can't graph trigonometric function of secx, cscx, cotx, tanx,...

source  code:

class PlotFunctions(GraphScene):
    CONFIG = {
        "x_min" : -10,
        "x_max" : 10.3,
        "y_min" : -1.5,
        "y_max" : 1.5,
        "graph_origin" : ORIGIN ,
        "function_color" : RED ,
        "axes_color" : GREEN,
        "x_labeled_nums" :range(-10,12,2),

    }
    def construct(self):
        self.setup_axes(animate=True)
        func_graph=self.get_graph(self.func_to_graph,self.function_color)
        func_graph2=self.get_graph(self.func_to_graph2)
        vert_line = self.get_vertical_line_to_graph(TAU,func_graph,color=YELLOW)
        graph_lab = self.get_graph_label(func_graph, label = "\\cos(x)")
        graph_lab2=self.get_graph_label(func_graph2,label = "\\sin(x)", x_val=-10, direction=UP/2)
        two_pi = TexMobject("x = 2 \\pi")
        label_coord = self.input_to_graph_point(TAU,func_graph)
        two_pi.next_to(label_coord,RIGHT+UP)

        self.play(ShowCreation(func_graph),ShowCreation(func_graph2))
        self.play(ShowCreation(vert_line), ShowCreation(graph_lab), ShowCreation(graph_lab2),ShowCreation(two_pi))

    def func_to_graph(self,x):
        #return np.cos(x)
        return np.tan(x)

    def func_to_graph2(self,x):
        return np.sin(x)

I replaced "return np.cos(x)" to "return np.tan(x)"...i got this:
![image](https://user-images.githubusercontent.com/36161299/63267544-e140a700-c2c4-11e9-9164-a14d37ee8673.png)

and then I replaced "return np.cos(x)" to "return np.sec(x)/cot(x)/csc(x)"...i got this:
AttributeError: module 'numpy' has no attribute 'sec'...


Manim can't find my png file

I'm new to coding and am trying to learn manim, which I'm using on my macbook pro. I'm trying to create a scene where manim draws a png file I saved. I saved the png file as "shirt.png" in my manim folder. I then ran the following code:

```
from manimlib.imports import *

class OutFit(Scene):
	def construct(self):

		shirt = ImageMobject("shirt")

		self.play(Write(shirt))
```
I've looked up several ways of how to get manim to do images and some solutions, but since I'm pretty new at this I don't always understand the answers I've found from other people's issues or if it applies to mine. I keep getting this error response:

raise IOError("File {} not Found".format(file_name))
OSError: File shirt not Found

Any help is much appreciated.


What is VMobject exactly?

Can anyone explain what is the purpose of `VMobject` and how it differs from `Mobject`?

I am trying to make some `old_projects` work. For example, I had to change `PMobject` to inherit from `VMobject` instead of `Mobject` in order to fix `NumberLineScene`. I do not know if it is correct thing to do or how will it affect the other scripts because I am unable to find the fundamental differences between the two objects. The wiki does not explain a lot, so please tell some detailed information.

I dug commit histories and saw

> "Starting to vectorize all things"

 kind of commit messages when the `VMobject` class is added to the engine. What does it mean "Vectorize" in this context?


[Documentation]: Micro-agents

**What problem or use case are you trying to solve?**

Currently in the `openhands/agenthub/codeact_agent` directory, we have an implementation of micro agents, but this is not documented.

To do so, we can:
1. read the implementation of codeact agent
2. read an example microagent in `openhands/agenthub/codeact_agent/micro/github.md`
3. add documentation to `openhands/agenthub/codeact_agent/README.md`


Ollama support issue.

<!-- You MUST fill out this template. We will close issues that don't include enough information to reproduce -->
#### Describe the bug

When trying to configure OpenDevin to run with Ollama there are requests that are being sent to the ollama server like this:

![image](https://github.com/OpenDevin/OpenDevin/assets/76570167/1931e068-0341-429b-8c4e-0dd2da36f54c)

The post request should look like this:
`"POST /chat/completions HTTP/1.1"`

<!-- a short description of the problem -->

#### Setup and configuration
**Current version**:
<!-- run `git log -n 1` to see this -->
```bash
commit 5c640c99cafb3c718dad60f377f3a725a8bab1de (HEAD -> local-llm-flag, origin/main, origin/HEAD, main)
```

<!-- tell us everything about your environment -->
**My config.toml and environment vars** (be sure to redact API keys):
```toml
WORKSPACE_DIR="./workspace"
LLM_BASE_URL="http://localhost:8000"
LLM_MODEL="ollama/starcoder2:15b"
LLM_EMBEDDING_MODEL="ollama/starcoder2:15b"
```

**My model and agent** (you can see these settings in the UI):
* Model: ollama/starcoder2
* Agent: MonologueAgent

**Commands I ran to install and run OpenDevin**:
```
git clone ...
make build
make start-backend
make start-frontend
```

**Steps to Reproduce**:
1. In `opendevin/llm/llm.py` in `__init__` replace `self.model = model if model else DEFAULT_MODEL_NAME`  with `self.model_name = DEFAULT_MODEL_NAME`
2. Run your local model on litellm `litellm --model ollama/starcoder2:15b --port 8000`
3. Run `make build` then `make start-backend` and `make start-frontend`
4. Ask devin to do anything ex 'make a hello world script in python'
5. Observe 404 errors spammed in litellm server log

**Logs, error messages, and screenshots**:
This is a log from the backend server running from `make start-backend` steps 0-99 all look the same.
```
==============
STEP 99

PLAN:
please make a simple flask app that says hello world.
Traceback (most recent call last):
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 1436, in function_with_retries
    response = original_function(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 386, in _completion
    raise e
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 334, in _completion
    deployment = self.get_available_deployment(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 2313, in get_available_deployment
    raise ValueError(f"No healthy deployment available, passed model={model}")
ValueError: No healthy deployment available, passed model=ollama/starcoder2:15b

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/quimbo/OpenDevin/agenthub/monologue_agent/utils/monologue.py", line 31, in condense
    resp = llm.completion(messages=messages)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 328, in completion
    raise e
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 325, in completion
    response = self.function_with_fallbacks(**kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 1419, in function_with_fallbacks
    raise original_exception
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 1344, in function_with_fallbacks
    response = self.function_with_retries(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 1496, in function_with_retries
    raise e
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 1462, in function_with_retries
    response = original_function(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 386, in _completion
    raise e
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 334, in _completion
    deployment = self.get_available_deployment(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 2313, in get_available_deployment
    raise ValueError(f"No healthy deployment available, passed model={model}")
ValueError: No healthy deployment available, passed model=ollama/starcoder2:15b

ERROR:
Error condensing thoughts: No healthy deployment available, passed model=ollama/starcoder2:15b
Traceback (most recent call last):
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 1436, in function_with_retries
    response = original_function(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 386, in _completion
    raise e
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 334, in _completion
    deployment = self.get_available_deployment(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 2313, in get_available_deployment
    raise ValueError(f"No healthy deployment available, passed model={model}")
ValueError: No healthy deployment available, passed model=ollama/starcoder2:15b

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/quimbo/OpenDevin/agenthub/monologue_agent/utils/monologue.py", line 31, in condense
    resp = llm.completion(messages=messages)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 328, in completion
    raise e
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 325, in completion
    response = self.function_with_fallbacks(**kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 1419, in function_with_fallbacks
    raise original_exception
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 1344, in function_with_fallbacks
    response = self.function_with_retries(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 1496, in function_with_retries
    raise e
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 1462, in function_with_retries
    response = original_function(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 386, in _completion
    raise e
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 334, in _completion
    deployment = self.get_available_deployment(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/.local/share/virtualenvs/OpenDevin-thTG-Evv/lib/python3.11/site-packages/litellm/router.py", line 2313, in get_available_deployment
    raise ValueError(f"No healthy deployment available, passed model={model}")
ValueError: No healthy deployment available, passed model=ollama/starcoder2:15b

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/quimbo/OpenDevin/opendevin/controller/agent_controller.py", line 112, in step
    action = self.agent.step(self.state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/quimbo/OpenDevin/agenthub/monologue_agent/agent.py", line 153, in step
    self._add_event(prev_action.to_dict())
  File "/home/quimbo/OpenDevin/agenthub/monologue_agent/agent.py", line 96, in _add_event
    self.monologue.condense(self.llm)
  File "/home/quimbo/OpenDevin/agenthub/monologue_agent/utils/monologue.py", line 36, in condense
    raise RuntimeError(f"Error condensing thoughts: {e}")
RuntimeError: Error condensing thoughts: No healthy deployment available, passed model=ollama/starcoder2:15b

OBSERVATION:
Error condensing thoughts: No healthy deployment available, passed model=ollama/starcoder2:15b
Exited before finishing
```

#### Additional Context

Litellm for local models is expecting api calls in the following format:

![image](https://github.com/OpenDevin/OpenDevin/assets/76570167/67b10c26-a9e6-44a1-a79e-908fc7d3749f)

From: `http://localhost:8000/#/`

I know that the problem is whatever is managing the api calls is set to call `/api/generate/` because this is the convention, but for local server that is not supported. I do not know where to look to fix this, any ideas?

The server responds when I test it like this:
```
def query_local_llm(prompt, limit=TOKEN_LIMIT):
    # Replace with your actual server address and port
    url = "http://0.0.0.0:8000/chat/completions"
    payload = {
        "model": "ollama/mistral",
        "messages" : [{"content": prompt, "role": "user"}],
        "max_tokens": limit
    }
    response = requests.post(url, json=payload)
```
![image](https://github.com/OpenDevin/OpenDevin/assets/76570167/b9bae877-5bd4-4864-b672-9678bb9a294e)


can LinkExtractor extract scrapy.link with node info

the html is like below, i want to extract the link `/example/category/pg{page}/`, but the `scrapy.link` does not contains the node info(`currentPage` and `totalPage`), how can i extract the link with the node info

``` html
<div class="page-box">
     <div page-url="/example/category/pg{page}/"
             totalPage="35"
             currentPage="1"
     </div>
</div>
```


version : Scrapy 2.1.0

```
2020-05-11 20:05:28 [scrapy.core.engine] INFO: Spider opened
2020-05-11 20:05:28 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-05-11 20:05:28 [dy] INFO: Spider opened: dy
2020-05-11 20:05:28 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method MemoryUsage.engine_started of <scrapy.extensions.memusage.MemoryUsage object at 0x0000000004D3A358>>
Traceback (most recent call last):
  File "D:\microsoft\python37\lib\site-packages\scrapy\utils\defer.py", line 161, in maybeDeferred_coro
    result = f(*args, **kw)
  File "D:\microsoft\python37\lib\site-packages\pydispatch\robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "D:\microsoft\python37\lib\site-packages\scrapy\extensions\memusage.py", line 55, in engine_started
    self.crawler.stats.set_value('memusage/startup', self.get_virtual_size())
  File "D:\microsoft\python37\lib\site-packages\scrapy\extensions\memusage.py", line 48, in get_virtual_size
    size = self.resource.getrusage(self.resource.RUSAGE_SELF).ru_maxrss
AttributeError: module 'resource' has no attribute 'getrusage'
```

```
2020-05-11 20:05:43 [scrapy.core.engine] INFO: Closing spider (finished)
2020-05-11 20:05:43 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 6751,
 'downloader/request_count': 14,
 'downloader/request_method_count/GET': 14,
 'downloader/response_bytes': 12380415,
 'downloader/response_count': 14,
 'downloader/response_status_count/200': 10,
 'downloader/response_status_count/302': 4,
 'elapsed_time_seconds': 14.631021,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2020, 5, 11, 12, 5, 43, 378200),
 'item_scraped_count': 65,
 'log_count/DEBUG': 85,
 'log_count/ERROR': 1,
 'log_count/INFO': 9,
 'request_depth_max': 1,
 'response_received_count': 10,
 'scheduler/dequeued': 6,
 'scheduler/dequeued/memory': 6,
 'scheduler/enqueued': 6,
 'scheduler/enqueued/memory': 6,
 'start_time': datetime.datetime(2020, 5, 11, 12, 5, 28, 747179)}
2020-05-11 20:05:43 [scrapy.core.engine] INFO: Spider closed (finished)
2020-05-11 20:05:43 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method MemoryUsage.engine_stopped of <scrapy.extensions.memusage.MemoryUsage object at 0x0000000004D3A358>>
Traceback (most recent call last):
  File "D:\microsoft\python37\lib\site-packages\scrapy\utils\defer.py", line 161, in maybeDeferred_coro
    result = f(*args, **kw)
  File "D:\microsoft\python37\lib\site-packages\pydispatch\robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "D:\microsoft\python37\lib\site-packages\scrapy\extensions\memusage.py", line 70, in engine_stopped
    for tsk in self.tasks:
AttributeError: 'MemoryUsage' object has no attribute 'tasks'
```

(edited for text formatting)


2023 Ford Maverick Not Recognized

### Describe the bug

Car Not Recognized

Looks like all the values for firmware are the same as what is already in values.py

### Which car does this affect?

Ford Maverick 2023

### Provide a route where the issue occurs

66833387c2bbbca0|2023-09-27--21-13-05

### openpilot version

master-ci

### Additional info

`{'carParams': {'alternativeExperience': 1,
               'autoResumeSng': True,
               'carFingerprint': 'mock',
               'carFw': [{'address': 2016,
                          'brand': 'ford',
                          'bus': 1,
                          'ecu': 'engine',
                          'fwVersion': b'PZ6A-14C204-JE\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
                          'logging': False,
                          'obdMultiplexing': True,
                          'request': [b'>\x00', b'"\xf1\x88'],
                          'responseAddress': 2024,
                          'subAddress': 0},
                         {'address': 1840,
                          'brand': 'ford',
                          'bus': 0,
                          'ecu': 'eps',
                          'fwVersion': b'NZ6C-14D003-AL\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
                          'logging': False,
                          'obdMultiplexing': True,
                          'request': [b'>\x00', b'"\xf1\x88'],
                          'responseAddress': 1848,
                          'subAddress': 0},
                         {'address': 1888,
                          'brand': 'ford',
                          'bus': 0,
                          'ecu': 'abs',
                          'fwVersion': b'PZ6C-2D053-ED\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
                          'logging': False,
                          'obdMultiplexing': True,
                          'request': [b'>\x00', b'"\xf1\x88'],
                          'responseAddress': 1896,
                          'subAddress': 0},
                         {'address': 1798,
                          'brand': 'ford',
                          'bus': 0,
                          'ecu': 'fwdCamera',
                          'fwVersion': b'NZ6T-14F397-AC\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
                          'logging': False,
                          'obdMultiplexing': True,
                          'request': [b'>\x00', b'"\xf1\x88'],
                          'responseAddress': 1806,
                          'subAddress': 0},
                         {'address': 1842,
                          'brand': 'ford',
                          'bus': 0,
                          'ecu': 'shiftByWire',
                          'fwVersion': b'NZ6P-14G395-AD\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
                          'logging': True,
                          'obdMultiplexing': True,
                          'request': [b'>\x00', b'"\xf1\x88'],
                          'responseAddress': 1850,
                          'subAddress': 0},
                         {'address': 1892,
                          'brand': 'ford',
                          'bus': 0,
                          'ecu': 'fwdRadar',
                          'fwVersion': b'NZ6T-14D049-AA\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
                          'logging': False,
                          'obdMultiplexing': True,
                          'request': [b'>\x00', b'"\xf1\x88'],
                          'responseAddress': 1900,
                          'subAddress': 0},
                         {'address': 2016,
                          'brand': 'mazda',
                          'bus': 1,
                          'ecu': 'engine',
                          'fwVersion': b'PZ6A-14C204-JE\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
                          'logging': False,
                          'obdMultiplexing': True,
                          'request': [b'"\xf1\x88'],
                          'responseAddress': 2024,
                          'subAddress': 0},
                         {'address': 1840,
                          'brand': 'mazda',
                          'bus': 0,
                          'ecu': 'eps',
                          'fwVersion': b'NZ6C-14D003-AL\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
                          'logging': True,
                          'obdMultiplexing': True,
                          'request': [b'"\xf1\x88'],
                          'responseAddress': 1848,
                          'subAddress': 0},
                         {'address': 1888,
                          'brand': 'mazda',
                          'bus': 0,
                          'ecu': 'abs',
                          'fwVersion': b'PZ6C-2D053-ED\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
                          'logging': True,
                          'obdMultiplexing': True,
                          'request': [b'"\xf1\x88'],
                          'responseAddress': 1896,
                          'subAddress': 0},
                         {'address': 1798,
                          'brand': 'mazda',
                          'bus': 0,
                          'ecu': 'fwdCamera',
                          'fwVersion': b'NZ6T-14F397-AC\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
                          'logging': True,
                          'obdMultiplexing': True,
                          'request': [b'"\xf1\x88'],
                          'responseAddress': 1806,
                          'subAddress': 0},
                         {'address': 1892,
                          'brand': 'mazda',
                          'bus': 0,
                          'ecu': 'fwdRadar',
                          'fwVersion': b'NZ6T-14D049-AA\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
                          'logging': True,
                          'obdMultiplexing': True,
                          'request': [b'"\xf1\x88'],
                          'responseAddress': 1900,
                          'subAddress': 0}],
               'carName': 'mock',
               'carVin': '3FTTW8E31PRA79783',
               'centerToFront': 1.350000023841858,
               'communityFeatureDEPRECATED': False,
               'dashcamOnly': False,
               'directAccelControlDEPRECATED': False,
               'enableApgsDEPRECATED': False,
               'enableBsm': False,
               'enableCameraDEPRECATED': False,
               'enableDsu': False,
               'enableGasInterceptor': False,
               'experimentalLongitudinalAvailable': False,
               'fingerprintSource': 'can',
               'flags': 0,
               'fuzzyFingerprint': False,
               'hasStockCameraDEPRECATED': False,
               'isPandaBlackDEPRECATED': False,
               'lateralTuning': {'pid': {'kf': 0.0}},
               'longitudinalActuatorDelayLowerBound': 0.15000000596046448,
               'longitudinalActuatorDelayUpperBound': 0.15000000596046448,
               'longitudinalTuning': {'deadzoneBP': [0.0], 'deadzoneV': [0.0], 'kf': 1.0, 'kiBP': [0.0], 'kiV': [1.0], 'kpBP': [0.0], 'kpV': [1.0]},
               'mass': 1836.0,
               'maxLateralAccel': 10.0,
               'maxSteeringAngleDegDEPRECATED': 0.0,
               'minEnableSpeed': -1.0,
               'minSpeedCanDEPRECATED': 0.0,
               'minSteerSpeed': 0.0,
               'networkLocation': 'fwdCamera',
               'notCar': False,
               'openpilotLongitudinalControl': False,
               'pcmCruise': True,
               'radarTimeStep': 0.05000000074505806,
               'radarUnavailable': False,
               'rotationalInertia': 3139.534912109375,
               'safetyConfigs': [{'safetyModel': 'noOutput', 'safetyParam': 0, 'safetyParam2DEPRECATED': 0, 'safetyParamDEPRECATED': 0}],
               'safetyModelDEPRECATED': 'silent',
               'safetyModelPassiveDEPRECATED': 'silent',
               'safetyParamDEPRECATED': 0,
               'startAccel': 0.0,
               'startingAccelRateDEPRECATED': 0.0,
               'startingState': False,
               'steerActuatorDelay': 0.0,
               'steerControlType': 'torque',
               'steerLimitAlert': False,
               'steerLimitTimer': 1.0,
               'steerRateCostDEPRECATED': 0.0,
               'steerRatio': 13.0,
               'steerRatioRear': 0.0,
               'stopAccel': -2.0,
               'stoppingControl': True,
               'stoppingDecelRate': 0.800000011920929,
               'tireStiffnessFactor': 1.0,
               'tireStiffnessFront': 201087.203125,
               'tireStiffnessRear': 317877.90625,
               'transmissionType': 'unknown',
               'vEgoStarting': 0.5,
               'vEgoStopping': 0.5,
               'wheelSpeedFactor': 1.0,
               'wheelbase': 2.700000047683716},
 'logMonoTime': 971923210573,
 'valid': True}
{'carParams': {'alternativeExperience': 1,
               'autoResumeSng': True,
               'carFingerprint': 'mock',
               'carFw': [{'address': 2016,
                          'brand': 'ford',
                          'bus': 1,
                          'ecu': 'engine',
                          'fwVersion': b'PZ6A-14C204-JE\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
                          'logging': False,
                          'obdMultiplexing': True,
                          'request': [b'>\x00', b'"\xf1\x88'],
                          'responseAddress': 2024,
                          'subAddress': 0},
                         {'address': 1840,
                          'brand': 'ford',
                          'bus': 0,
                          'ecu': 'eps',
                          'fwVersion': b'NZ6C-14D003-AL\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
                          'logging': False,
                          'obdMultiplexing': True,
                          'request': [b'>\x00', b'"\xf1\x88'],
                          'responseAddress': 1848,
                          'subAddress': 0},
                         {'address': 1888,
                          'brand': 'ford',
                          'bus': 0,
                          'ecu': 'abs',
                          'fwVersion': b'PZ6C-2D053-ED\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
                          'logging': False,
                          'obdMultiplexing': True,
                          'request': [b'>\x00', b'"\xf1\x88'],
                          'responseAddress': 1896,
                          'subAddress': 0},
                         {'address': 1798,
                          'brand': 'ford',
                          'bus': 0,
                          'ecu': 'fwdCamera',
                          'fwVersion': b'NZ6T-14F397-AC\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
                          'logging': False,
                          'obdMultiplexing': True,
                          'request': [b'>\x00', b'"\xf1\x88'],
                          'responseAddress': 1806,
                          'subAddress': 0},
                         {'address': 1842,
                          'brand': 'ford',
                          'bus': 0,
                          'ecu': 'shiftByWire',
                          'fwVersion': b'NZ6P-14G395-AD\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
                          'logging': True,
                          'obdMultiplexing': True,
                          'request': [b'>\x00', b'"\xf1\x88'],
                          'responseAddress': 1850,
                          'subAddress': 0},
                         {'address': 1892,
                          'brand': 'ford',
                          'bus': 0,
                          'ecu': 'fwdRadar',
                          'fwVersion': b'NZ6T-14D049-AA\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
                          'logging': False,
                          'obdMultiplexing': True,
                          'request': [b'>\x00', b'"\xf1\x88'],
                          'responseAddress': 1900,
                          'subAddress': 0},
                         {'address': 2016,
                          'brand': 'mazda',
                          'bus': 1,
                          'ecu': 'engine',
                          'fwVersion': b'PZ6A-14C204-JE\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
                          'logging': False,
                          'obdMultiplexing': True,
                          'request': [b'"\xf1\x88'],
                          'responseAddress': 2024,
                          'subAddress': 0},
                         {'address': 1840,
                          'brand': 'mazda',
                          'bus': 0,
                          'ecu': 'eps',
                          'fwVersion': b'NZ6C-14D003-AL\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
                          'logging': True,
                          'obdMultiplexing': True,
                          'request': [b'"\xf1\x88'],
                          'responseAddress': 1848,
                          'subAddress': 0},
                         {'address': 1888,
                          'brand': 'mazda',
                          'bus': 0,
                          'ecu': 'abs',
                          'fwVersion': b'PZ6C-2D053-ED\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
                          'logging': True,
                          'obdMultiplexing': True,
                          'request': [b'"\xf1\x88'],
                          'responseAddress': 1896,
                          'subAddress': 0},
                         {'address': 1798,
                          'brand': 'mazda',
                          'bus': 0,
                          'ecu': 'fwdCamera',
                          'fwVersion': b'NZ6T-14F397-AC\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
                          'logging': True,
                          'obdMultiplexing': True,
                          'request': [b'"\xf1\x88'],
                          'responseAddress': 1806,
                          'subAddress': 0},
                         {'address': 1892,
                          'brand': 'mazda',
                          'bus': 0,
                          'ecu': 'fwdRadar',
                          'fwVersion': b'NZ6T-14D049-AA\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
                          'logging': True,
                          'obdMultiplexing': True,
                          'request': [b'"\xf1\x88'],
                          'responseAddress': 1900,
                          'subAddress': 0}],
               'carName': 'mock',
               'carVin': '3FTTW8E31PRA79783',
               'centerToFront': 1.350000023841858,
               'communityFeatureDEPRECATED': False,
               'dashcamOnly': False,
               'directAccelControlDEPRECATED': False,
               'enableApgsDEPRECATED': False,
               'enableBsm': False,
               'enableCameraDEPRECATED': False,
               'enableDsu': False,
               'enableGasInterceptor': False,
               'experimentalLongitudinalAvailable': False,
               'fingerprintSource': 'can',
               'flags': 0,
               'fuzzyFingerprint': False,
               'hasStockCameraDEPRECATED': False,
               'isPandaBlackDEPRECATED': False,
               'lateralTuning': {'pid': {'kf': 0.0}},
               'longitudinalActuatorDelayLowerBound': 0.15000000596046448,
               'longitudinalActuatorDelayUpperBound': 0.15000000596046448,
               'longitudinalTuning': {'deadzoneBP': [0.0], 'deadzoneV': [0.0], 'kf': 1.0, 'kiBP': [0.0], 'kiV': [1.0], 'kpBP': [0.0], 'kpV': [1.0]},
               'mass': 1836.0,
               'maxLateralAccel': 10.0,
               'maxSteeringAngleDegDEPRECATED': 0.0,
               'minEnableSpeed': -1.0,
               'minSpeedCanDEPRECATED': 0.0,
               'minSteerSpeed': 0.0,
               'networkLocation': 'fwdCamera',
               'notCar': False,
               'openpilotLongitudinalControl': False,
               'pcmCruise': True,
               'radarTimeStep': 0.05000000074505806,
               'radarUnavailable': False,
               'rotationalInertia': 3139.534912109375,
               'safetyConfigs': [{'safetyModel': 'noOutput', 'safetyParam': 0, 'safetyParam2DEPRECATED': 0, 'safetyParamDEPRECATED': 0}],
               'safetyModelDEPRECATED': 'silent',
               'safetyModelPassiveDEPRECATED': 'silent',
               'safetyParamDEPRECATED': 0,
               'startAccel': 0.0,
               'startingAccelRateDEPRECATED': 0.0,
               'startingState': False,
               'steerActuatorDelay': 0.0,
               'steerControlType': 'torque',
               'steerLimitAlert': False,
               'steerLimitTimer': 1.0,
               'steerRateCostDEPRECATED': 0.0,
               'steerRatio': 13.0,
               'steerRatioRear': 0.0,
               'stopAccel': -2.0,
               'stoppingControl': True,
               'stoppingDecelRate': 0.800000011920929,
               'tireStiffnessFactor': 1.0,
               'tireStiffnessFront': 201087.203125,
               'tireStiffnessRear': 317877.90625,
               'transmissionType': 'unknown',
               'vEgoStarting': 0.5,
               'vEgoStopping': 0.5,
               'wheelSpeedFactor': 1.0,
               'wheelbase': 2.700000047683716},
 'logMonoTime': 1021914306894,
 'valid': True}`


Content marked as consumed in 0.13.6

Content is immediately marked as consumed in 0.13.6, causing calls to e.g. response.iter_content() to throw an error.

Test code (tested with python 2.6):

```
import requests

r = requests.get('http://docs.python-requests.org/')
if r._content_consumed:
    print 'consumed'
else:
    print 'not consumed'
```

In 0.13.5 this prints:
not consumed

In 0.13.6 this prints:
consumed


Chunk-encoded request doesn't recognize iter_content generator

Passing a generator created by iter_content() as request data raises "TypeError: sendall() argument 1 must be string or buffer, not generator".

## Expected Result

The POST request successfully delives the content from the GET request.

## Actual Result

A TypeError is raised:
```
Traceback (most recent call last):
  File "..\test.py", line 7, in <module>
    PostForward("http://myhost/img/foo.png", "http://myotherhost/convert")
  File "..\test.py", line 6, in PostForward
    return requests.post(url=dst, data=data, headers={'Content-Length': length})
  File "C:\Python27\lib\site-packages\requests\api.py", line 112, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "C:\Python27\lib\site-packages\requests\api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python27\lib\site-packages\requests\adapters.py", line 440, in send
    timeout=timeout
  File "C:\Python27\lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "C:\Python27\lib\site-packages\urllib3\connectionpool.py", line 357, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "C:\Python27\lib\httplib.py", line 1042, in request
    self._send_request(method, url, body, headers)
  File "C:\Python27\lib\httplib.py", line 1082, in _send_request
    self.endheaders(body)
  File "C:\Python27\lib\httplib.py", line 1038, in endheaders
    self._send_output(message_body)
  File "C:\Python27\lib\httplib.py", line 886, in _send_output
    self.send(message_body)
  File "C:\Python27\lib\httplib.py", line 858, in send
    self.sock.sendall(data)
  File "C:\Python27\lib\socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
TypeError: sendall() argument 1 must be string or buffer, not generator
```

## Reproduction Steps

```python
import requests
def PostForward(src, dst):
	with requests.get(url=src, stream=True) as srcResponse:
		length = srcResponse.headers['Content-Length']
		data = srcResponse.iter_content(1024)
		return requests.post(url=dst, data=data, headers={'Content-Length': length})
PostForward("http://myhost/img/foo.png", "http://myotherhost/convert")
```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "2.7.14"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.18.4"
  },
  "system_ssl": {
    "version": "100020bf"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": false
}
```


[WinError 10048] Only one usage of each socket address ...

I notice that despite using requests.Session() - I still seem to be creating new connections/sockets which eventually exhaust (TIME_WAIT) and I get the following error:

> [WinError 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted',))

```
s = requests.Session()
data = zip(url_routes, cycle(s))
calc_routes = pool.map(processRequest, data)

```

I posted a bit more [here](http://stackoverflow.com/questions/35793908/python-multiprocessing-associate-a-process-with-a-session), however not sure how to address this


File download weirdness

I noticed this issue building conda recipes.  Conda uses requests to download files from the internet.

The file that is being fetched is: https://dakota.sandia.gov/sites/default/files/distributions/public/dakota-6.5-public.src.tar.gz
(link found here: https://dakota.sandia.gov/download.html)

Downloading with curl -O
filesize: 78MB
md5: 02c46e904d40bba6b308065db34c1ad7

Downloading with urllib2 (from the standard library):
filesize: 78MB
md5: 02c46e904d40bba6b308065db34c1ad7

Downloading with requests-2.12.1 (supplied with conda)
filesize: 248MB
md5: 41e4268140d850756812510512d8eee8
tar -tf doesn't indicate any corruption.

I'm not sure what is different with this particular URL, but the other  files I tried with requests worked.  I don't know where the extra 170MB is coming from?

code used to download files:
```python
def download_file(url, fn):
    r = requests.get(url, stream=True)
    with open(fn, 'wb') as f:
        for chunk in r.iter_content(chunk_size=1024):
            if chunk:
                f.write(chunk)

def download_urllib2(url, fn):
    f = urllib2.urlopen(url)
    with open(fn, 'wb') as fh:
        for x in iter(lambda: f.read(1024), b''):
            fh.write(x)
```


```python
import requests

requests.get('http://gett.bike/')
```
This code raises the following exception:
```python
ContentDecodingError: ('Received response with content-encoding: gzip, but failed to decode it.',
error('Error -3 while decompressing data: incorrect data check',))
```
Arch linux x64
requests==2.13.0
python=3.6.0


Ability to set timeout after response

For devs who use this great library, it would be very beneficial to be able to set the timeout AFTER initial connection. There are a few scenarios where this is useful but one of the main patterns/use cases is this:

```

import requests
import socket

# May or may not subclass threading.Thread
class Getter(object):
    def __init__(self):
        self.request = requests.get(url, stream=True)

    def run(self):
        with open(path, 'r+b') as file:

            bytes_consumed = 0
            while True:
                try:

                    chunk = self.request.raw.read(size)
                    if not chunk:
                        break
                    chunk_length = len(chunk)

                    file.write(chunk)
                    bytes_consumed += chunk_length

                except socket.timeout:
                    # handle incomplete download by using range header next time, etc.
```

Handling incomplete downloads due to connection loss is common and especially important when downloading large or many files (or both). As you can see, this can be achieved in a fairly straightforward way. The issue is there is really no good way to write tests for this. Each method would involve OS specific code which would also be a no-go for CI services.

What would be an option is the ability to set the timeout after establishing a connection. This way in a test you could do "r.timeout = (None, 0.00001)" and during reading it would simulate a timeout.

To my knowledge this is no way currently to inject a new Timeout class retroactively. Is this correct?


bad handshake error with ssl3

I have an inhouse IIS server with ssl3 but an expired certificate, so I used requests without certificate verification and it was working fine with requests 2.11.1. But after I upgrade requests to 2.12.0, there was an error occured.

the code is:
...
requests.get('https://10.192.8.89:8080/yps_report', verify=False)
...

error message:
Traceback (most recent call last):
  File "c:\python35\lib\site-packages\requests\packages\urllib3\contrib\pyopenssl.py", line 417, in wrap_socket
    cnx.do_handshake()
  File "c:\python35\lib\site-packages\OpenSSL\SSL.py", line 1426, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "c:\python35\lib\site-packages\OpenSSL\SSL.py", line 1167, in _raise_ssl_error
    raise SysCallError(-1, "Unexpected EOF")
OpenSSL.SSL.SysCallError: (-1, 'Unexpected EOF')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\python35\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 594, in urlopen
    chunked=chunked)
  File "c:\python35\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 350, in _make_request
    self._validate_conn(conn)
  File "c:\python35\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 835, in _validate_conn
    conn.connect()
  File "c:\python35\lib\site-packages\requests\packages\urllib3\connection.py", line 323, in connect
    ssl_context=context)
  File "c:\python35\lib\site-packages\requests\packages\urllib3\util\ssl_.py", line 324, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "c:\python35\lib\site-packages\requests\packages\urllib3\contrib\pyopenssl.py", line 424, in wrap_socket
    raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: SysCallError(-1, 'Unexpected EOF')",)
...

I tried  to downgrade requests to 2.11.1 and the error was gone. I have no idea how to fix this.
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.ssl_ import create_urllib3_context

# This is the 2.11 Requests cipher string.
CIPHERS = (
    'ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:'
    'DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:!aNULL:'
    '!eNULL:!MD5'
)

class DESAdapter(HTTPAdapter):
    def init_poolmanager(self, *args, **kwargs):
        context = create_urllib3_context(ciphers=CIPHERS)
        kwargs['ssl_context'] = context
        return super(HTTPAdapter, self).init_poolmanager(*args, **kwargs)

s = requests.Session()
s.mount('https://10.192.8.89', DESAdapter())

3
需要将下面的user的一个comment中user的代码放入其中


### Summary

When trying to pass a variable called i.e. sysctl.values to loop, I will get the above error.

### Issue Type

Bug Report

### Component Name

debug (only used for debugging)

### Ansible Version

```console
$ ansible --version
ansible 2.9.27
  config file = /home/rf/.ansible.cfg
  configured module search path = ['/home/rf/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.10/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.10.6 (main, Aug  2 2022, 00:00:00) [GCC 11.3.1 20220421 (Red Hat 11.3.1-2)]
```

### Configuration

```console
# if using a version older than ansible-core 2.12 you should omit the '-t all'
$ ansible-config dump --only-changed -t all

[I] </m/d/playground>-2-> ansible-config dump --only-changed
ANSIBLE_PIPELINING(/home/rf/.ansible.cfg) = True
ANSIBLE_SSH_ARGS(/home/rf/.ansible.cfg) = -o ControlMaster=auto -o ControlPersist=60s
DEFAULT_FORKS(/home/rf/.ansible.cfg) = 50
DEFAULT_HOST_LIST(/home/rf/.ansible.cfg) = ['/home/rf/hosts']
INVENTORY_CACHE_ENABLED(/home/rf/.ansible.cfg) = True
```

### OS / Environment

Fedora 36

### Steps to Reproduce

<!--- Paste example playbooks or commands between quotes below -->
```yaml (paste below)
- name: Test
  hosts: localhost
  gather_facts: True
  tasks:
    - debug:
        msg: "{{ item }}"
      loop: "{{ sysctl2 }}"
    - debug:
        msg: "{{ item }}"
      loop: "{{ sysctl.values }}"
  vars:
    sysctl:
      values:
        - { name: "net.ipv4.ip_forward", value: "1" }
    sysctl2:
        - { name: "net.ipv4.ip_forward", value: "1" }
```

### Expected Results

Output of debug using sysctl.values

### Actual Results

```console
PLAY [Test] ********************************************************************************************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************************************************************************
ok: [localhost]

TASK [debug] *******************************************************************************************************************************************************************************************
ok: [localhost] => (item={'name': 'net.ipv4.ip_forward', 'value': '1'}) => {
    "msg": {
        "name": "net.ipv4.ip_forward",
        "value": "1"
    }
}

TASK [debug] *******************************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"msg": "Invalid data passed to 'loop', it requires a list, got this instead: <built-in method values of dict object at 0x7f63b782bf80>. Hint: If you passed a list/dict of just one element, try adding wantlist=True to your lookup invocation or use q/query instead of lookup."}

PLAY RECAP *********************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
```
```

### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct


Ansible core 2.16.1 broke AnsibleUnsafeBytes iteration

### Summary

Upgrading form 2.16.0 to 2.16.1 (Ansible 9.0.1 to 9.1.0), iterating over AnsibleUnsafeBytes does not create a list of numbers anymore.

### Issue Type

Bug Report

### Component Name

core, unsafe_proxy

### Ansible Version

```console
$ ansible --version

ansible [core 2.16.1]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.12/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.12.0 (main, Nov 29 2023, 03:32:06) [GCC 10.2.1 20210110] (/usr/local/bin/python)
  jinja version = 3.1.2
  libyaml = True
```

### Configuration

```console
# if using a version older than ansible-core 2.12 you should omit the '-t all'
$ ansible-config dump --only-changed -t all

/bin/sh: 1: less: not found
```

(sorry, dockerized environment)

### OS / Environment

Debian bullseye / 11 (in python docker image: `python:3.12.0-bullseye`), ansible via pip (`ansible==9.1.0`)

### Steps to Reproduce

<!--- Paste example playbooks or commands between quotes below -->
```py
from ansible.utils.unsafe_proxy import AnsibleUnsafeText
x = AnsibleUnsafeText("asdf")
y = x.encode("utf8")
list(y)
```

### Expected Results

```
[97, 115, 100, 102]
```

This is what happens on 2.16.0.

### Actual Results

```console
[b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00']
```

### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct


docker_container task fail on exit code

Unless i'm missing something i expect that if I were to do something like the following the task would fail? But it does not 😟

```yaml
  tasks:
      docker_container:
        name: "exit-test"
        image: "ubuntu:latest"
        command: "bash -c 'exit 123'"
```

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
docker_container

##### ANSIBLE VERSION
```
2.1.1.0
```

##### OS / ENVIRONMENT
N/A

##### STEPS TO REPRODUCE
```yaml
  tasks:
      docker_container:
        name: "exit-test"
        image: "ubuntu:latest"
        command: "bash -c 'exit 123'"
```
##### EXPECTED RESULTS
Should fail the task

##### ACTUAL RESULTS
Task is ok.


affects_2.0
module
support:core
bug
files

Template update convert \n to actual new line

##### ISSUE TYPE

    Bug Report

##### COMPONENT NAME

template

##### ANSIBLE VERSION

2.0 and higher
CONFIGURATION
```
[ssh_connection]
control_path = %(directory)s/%%C
```
##### OS / ENVIRONMENT

Mac OS X 10.11.6
Centos 6.x, 7.x
SUMMARY

In the input .j2 file, we substitute a variable with an environment variable that has a line/string that contains a grok expression containing `(?m)\n` . The output generated by the template module in versions 2.0 and later, treats the \n as actual line break. Where as versions up to 1.9.6 retains the literal `(?m)\n` without replacing the \n with an actual line break. We see the line break after we upgraded the Ansible version to 2.x.

Any way we can work around this issue? Thank you for your help.
##### STEPS TO REPRODUCE

Our execution flow is probably not the nicest - we want to reengineer it soon. Basic steps:

    Run a shell script with ansible-playbook command that pass in an env variable with `(?m)\n` literal.
    Playbook calls a main yaml file and assigns shell environment var to a included task yaml file.
    The task yaml file invokes the template module.

In the snippet below I stripped out other lines/vars for clarity.

main shell
```
set GROK_PATTERN_GENERAL_ERROR_PG="%{TIMESTAMP_ISO8601} ERROR \[%{USER:handlerName}\] %{USER:className}%{GREEDYDATA:errorline1}((?m)\n%{USER:logerror}%{GREEDYDATA})"
```
```
ansible-playbook -i ../common/host.inventory \
                -${VERBOSE} \
                t.yml \
                ${CHECK_ONLY} \
                --extra-vars "hosts='${HOST}'
                              xlogstash_grok_general_error='${GROK_PATTERN_GENERAL_ERROR_PG}'
                             "
```
t.yml
```
---
- hosts: 127.0.0.1
  connection: local

  tasks:
    - include_vars: ../common/defaults/main.yml
    - name: generate logstash kafka logscan filter config file
      include: tasks/t.yml
      vars:
          logstash_grok_general_error: "{{xlogstash_grok_general_error}}"
```
tasks/t.yml
```
---
    - name: generate logstash kafka logscan filter config file
      template: src=../common/templates/my.conf.j2
                dest="./500-filter.conf"
```
my.conf.j2
```
        grok {
                break_on_match => "true"
                match => [
                        "message", "{{logstash_grok_general_error}}"
                ]
        }
```
Note the `(?m)\n` are still on the same line.
##### EXPECTED RESULTS
```
        grok {
                break_on_match => "true"
                match => [
                        "message", "%{TIMESTAMP_ISO8601} ERROR \[%{USER:handlerName}\] %{USER:className}%{GREEDYDATA:errorline1}((?m)\n%{USER:logerror}%{GREEDYDATA})"
                ]
        }
```
##### ACTUAL RESULTS

Note `(?m)\n` now has the `\n` as actual line break.
```
        grok {
                break_on_match => "true"
                match => [
                        "message", "%{TIMESTAMP_ISO8601} ERROR \[%{USER:handlerName}\] %{USER:className}%{GREEDYDATA:errorline1}((?m)
%{USER:logerror}%{GREEDYDATA})"
                ]
        }
```


python3
module
support:core
bug
affects_2.10

cron: Remove/delete an environment variable

### Summary

With `env=yes`, `cron` add environment variable (with the `name` & `value`) parameters.
I though that having `env` + `state=absent` would remove said variable, but that's not the case (the cron file is actually removed).
As such there is no way to remove a variable and the more obvious way to attempt to do it results in a surprising result.

### Issue Type

Bug Report

### Component Name

ansible.builtin.cron

### Ansible Version

```console
$ ansible --version
ansible 2.10.5
  config file = /home/user/.ansible.cfg
  configured module search path = ['/usr/share/ansible']
  ansible python module location = /home/user/.local/lib/python3.8/site-packages/ansible
  executable location = /home/user/.local/bin/ansible
  python version = 3.8.5 (default, Jan 27 2021, 15:41:15) [GCC 9.3.0]

```

### Configuration

```console (paste below)
$ ansible-config dump --only-changed

```

### OS / Environment

Ubuntu 20.04

### Steps to Reproduce

```yaml
  cron:
    cron_file: foobar
    user: root
    env: yes
    name: "VAR"
    value: "False"
    state: absent
```

### Expected Results

The "VAR" variable is removed from /etc/cron.d/foobar

### Actual Results

/etc/cron.d/foobar is removed.
There is no way to remove the "VAR" variable.


affects_2.0
affects_2.3
c:executor/playbook_executor
support:core
feature
P3

enable do/until with async tasks

##### ISSUE TYPE
Feature Idea

##### COMPONENT NAME
core

##### ANSIBLE VERSION
2.0

##### CONFIGURATION

##### OS / ENVIRONMENT

##### SUMMARY
When a task is marked as async, there is no way to loop until a condition is met.
With poll:0 and async_status you can poll for async task to complete but you cannot repeat the original async task itself until a condition is met.

```
cat /tmp/async-test.yml

---
# Run through the test of an async command

- hosts: all
  tasks:
    - name: "Check an async command"
      command: /bin/sleep 3
      async: 5
      poll: 1
      register: command_result
      until: command_result.failed
      retries: 5
      delay: 10
```

```
$ansible-playbook -i localhost, /tmp/async-test.yml
 ____________
< PLAY [all] >
 ------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

 _________________
< GATHERING FACTS >
 -----------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

ok: [localhost]
 ______________________________
< TASK: Check an async command >
 ------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

fatal: [localhost] => error while evaluating conditional: command_result.failed: {% if command_result.failed %} True {% else %} False {% endif %}

FATAL: all hosts have already failed -- aborting
 ____________
< PLAY RECAP >
 ------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

           to retry, use: --limit @/opt/ashishkh/async-test.retry

localhost                  : ok=1    changed=0    unreachable=2    failed=0
```

##### STEPS TO REPRODUCE

##### EXPECTED RESULTS

##### ACTUAL RESULTS


cloud
aws
openstack
module
support:community
affects_2.5
bug
traceback
system

Stickiness type 'lb_cookie' is not supported for target groups with the TCP protocol(unable to disable stickiness not supported in NLB)

##### SUMMARY
We are using Ansible 2.5 to deploy AWS resources in our environment. From March 02, 2019 our deployment is failing with the below error.

ERROR:
=====
TASK [immutable_server : target group for analytics-tst-plebos loadbalancer] ***
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: InvalidConfigurationRequestException:
An error occurred (InvalidConfigurationRequest) when calling the ModifyTargetGroupAttributes operation:
Stickiness type 'lb_cookie' is not supported for target groups with the TCP protocol
17:21:08 fatal: [localhost]: FAILED! => {"changed": false, "error": {"code": "InvalidConfigurationRequest", "message": "Stickiness type 'lb_cookie'
is not supported for target groups with the TCP protocol", "type": "Sender"}, "msg": "An error occurred (InvalidConfigurationRequest)
when calling the ModifyTargetGroupAttributes operation: Stickiness type 'lb_cookie' is not supported for target groups with the TCP protocol",
"response_metadata": {"http_headers": {"connection": "close", "content-length": "359", "content-type": "text/xml", "date": "Tue, 03 Mar 2020 11:51:08 GMT",
"x-amzn-requestid": "23b0ca87-e0fb-4b84-b93b-ae5b1363df53"}, "http_status_code": 400, "request_id": "23b0ca87-e0fb-4b84-b93b-ae5b1363df53", "retry_attempts": 0}}

##### ISSUE TYPE
- Bug Report - Unable to disable stickiness not supported in NLB

##### COMPONENT NAME
- name: "target group for {{ server_name }} loadbalancer"
  elb_target_group:
    state: present
    name: "{{ server_name }}-elb"
    protocol: tcp
    port: 80
    target_type: instance
    deregistration_delay_timeout: 35
    modify_targets: False
    vpc_id: "{{ vpc_out.vpcs.0.id }}"
    health_check_protocol: "{{ load_balancer_ping_protocol | default('http') }}"
    health_check_port: "{{ load_balancer_ping_port | default('80') }}"
    health_check_path: "{{ load_balancer_ping_path | default('/elb/ping')}}"
    health_check_interval: 30
    unhealthy_threshold_count: 2
    healthy_threshold_count: 2
    stickiness_enabled: False
    tags: "{{ aws.tags_as_dict }}"
  register: target_group_out

##### ANSIBLE VERSION
```paste below
Ansible version = 2.5.0
```

##### CONFIGURATION
<!--- Paste verbatim output from "ansible-config dump --only-changed" between quotes -->
```paste below
- name: "target group for {{ server_name }} loadbalancer"
  elb_target_group:
    state: present
    name: "{{ server_name }}-elb"
    protocol: tcp
    port: 80
    target_type: instance
    deregistration_delay_timeout: 35
    modify_targets: False
    vpc_id: "{{ vpc_out.vpcs.0.id }}"
    health_check_protocol: "{{ load_balancer_ping_protocol | default('http') }}"
    health_check_port: "{{ load_balancer_ping_port | default('80') }}"
    health_check_path: "{{ load_balancer_ping_path | default('/elb/ping')}}"
    health_check_interval: 30
    unhealthy_threshold_count: 2
    healthy_threshold_count: 2
    stickiness_enabled: False
    tags: "{{ aws.tags_as_dict }}"
  register: target_group_out

```

##### OS / ENVIRONMENT
Ubuntu 18.04 LTS / AWS environment

##### STEPS TO REPRODUCE
Kindly use the below playbook to deploy loadbalancer using Ansible on AWS cloud.

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- name: "target group for {{ server_name }} loadbalancer"
  elb_target_group:
    state: present
    name: "{{ server_name }}-elb"
    protocol: tcp
    port: 80
    target_type: instance
    deregistration_delay_timeout: 35
    modify_targets: False
    vpc_id: "{{ vpc_out.vpcs.0.id }}"
    health_check_protocol: "{{ load_balancer_ping_protocol | default('http') }}"
    health_check_port: "{{ load_balancer_ping_port | default('80') }}"
    health_check_path: "{{ load_balancer_ping_path | default('/elb/ping')}}"
    health_check_interval: 30
    unhealthy_threshold_count: 2
    healthy_threshold_count: 2
    stickiness_enabled: False
    tags: "{{ aws.tags_as_dict }}"
  register: target_group_out
```

<!--- HINT: You can paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
An AWS Network loadbalancer will be created.

##### ACTUAL RESULTS
The deployment fails with below error.

<!--- Paste verbatim command output between quotes -->
```paste below
 TASK [immutable_server : target group for analytics-tst-plebos loadbalancer] ***
17:21:08 An exception occurred during task execution. To see the full traceback, use -vvv. The error was: InvalidConfigurationRequestException:
An error occurred (InvalidConfigurationRequest) when calling the ModifyTargetGroupAttributes operation:
Stickiness type 'lb_cookie' is not supported for target groups with the TCP protocol
17:21:08 fatal: [localhost]: FAILED! => {"changed": false, "error": {"code": "InvalidConfigurationRequest", "message": "Stickiness type 'lb_cookie'
is not supported for target groups with the TCP protocol", "type": "Sender"}, "msg": "An error occurred (InvalidConfigurationRequest)
when calling the ModifyTargetGroupAttributes operation: Stickiness type 'lb_cookie' is not supported for target groups with the TCP protocol",
"response_metadata": {"http_headers": {"connection": "close", "content-length": "359", "content-type": "text/xml", "date": "Tue, 03 Mar 2020 11:51:08 GMT",
"x-amzn-requestid": "23b0ca87-e0fb-4b84-b93b-ae5b1363df53"}, "http_status_code": 400, "request_id": "23b0ca87-e0fb-4b84-b93b-ae5b1363df53", "retry_attempts": 0}}

```

##### References
I can see a similar issue occurred for terraform users as well.

https://github.com/terraform-providers/terraform-provider-aws/issues/10494


Problems with weights

## ❔Question
Hello, I have just run trainy.py script with my data and faced a problem - you wrote that weights are saved in runs directory, but in my case I have not found them. Everything is fine with hyp.yaml and opt.yaml but folder "weights" is empty.
Do you have any guesses about this issue?

## Additional context


Does adjusting the number of classes of a pretrained model work?

### Search before asking

- [x]  I have searched the YOLOv5 [issues](https://github.com/ultralytics/yolov5/issues) and [discussions](https://github.com/ultralytics/yolov5/discussions) and found no similar questions. *

### Question

Hi everyone,

I'm a bit confused about how to properly load a pretrained model with an adjusted number of classes for training with a custom dataset.

On the [Load YOLOv5 from PyTorch Hub ⭐](https://github.com/ultralytics/yolov5/issues/36) page you've explained that one can adjust  the  number of classes in the pretrained model by using the following command. `model = torch.hub.load('ultralytics/yolov5', 'yolov5s', classes=10)`

<img width="999" alt="Bildschirmfoto 2022-08-22 um 08 13 15" src="https://user-images.githubusercontent.com/5917496/185851461-b177aa78-2b56-46a1-9c43-081d2a746938.png">

When I do so, I can see that a model.yaml file is overwritten, but I do not know where this file is stored.

Now, what actually confuses me about the number of classes, is that when I try to use this pretrained model in detection, without any further training. I see an error, that the model was trained with nc=80 and my data is incompatible with nc=13:

`AssertionError: ['yolov5s6.pt'] (80 classes) trained on different --data than what you passed (13 classes). Pass correct combination of --weights and --data that are trained together.`

I know that I can not expect any proper predictions since the last layers are initialized with random weights, but I was expecting that the model is compatible with the 13 classes dataset.

Is this behavior to be expected or am I doing something wrong here?
Do I need to find and use the model.yaml file and is the only thing changed in there 'nc=13'?


training actually can not work

After trained on several epochs, I found the mAP is still very low. Does the training really works?

```
    Epoch   gpu_mem      GIoU       obj       cls     total   targets  img_size
    14/299      6.4G   0.02273  0.002925 0.0003764   0.02603        11       640: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6960/6960 [54:20<00:00,  2.13it/s]
               Class      Images     Targets           P           R      mAP@.5  mAP@.5:.95: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6960/6960 [13:37<00:00,  8.51it/s]
                 all    5.57e+04    1.74e+05    0.000332     0.00039     2.4e-06    8.59e-07

     Epoch   gpu_mem      GIoU       obj       cls     total   targets  img_size
    15/299      6.4G   0.02232  0.002874  0.000371   0.02556         7       640: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6960/6960 [54:36<00:00,  2.12it/s]
               Class      Images     Targets           P           R      mAP@.5  mAP@.5:.95: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6960/6960 [14:23<00:00,  8.06it/s]
                 all    5.57e+04    1.74e+05    0.000342    0.000401    2.44e-06    8.66e-07

```


training stuck when loading dataset

## ❔Question
I follow the instructions to run coco128,
```
python train.py --img 640 --batch 16 --epochs 5 --data ./data/coco128.yaml --cfg ./models/yolov5s.yaml --weights '',
```
the ouput is
```
Image sizes 640 train, 640 test
Using 8 dataloader workers
Starting training for 5 epochs...

     Epoch   gpu_mem      GIoU       obj       cls     total   targets  img_size
  0%|                                                                                | 0/8 [00:00<?, ?it/s
```
then it is stuck, I found that it is stucking at loading the dataset,
in https://github.com/ultralytics/yolov5/blob/master/train.py#L244,
```
for i, (imgs, targets, paths, _) in pbar:
```
it just stops here, could you help me ?


Results interpretation

Hello,

Another question to do with results interpretation. I am not very sure how to interpret the results.txt file that gets generated after training is over. Also, is there any way to extract the number of false positives, true positives, false negatives, as well as to see the total mean average accuracy and loss (like with yolov4)?

Further, after training is done, can the best weights obtained from training be used to test on unseen data (more specifically, multiple images)?

Thanks in advance again!


How to add some private information into .pt file?

### Search before asking

- [X] I have searched the YOLOv5 [issues](https://github.com/ultralytics/yolov5/issues) and [discussions](https://github.com/ultralytics/yolov5/discussions) and found no similar questions.

### Question

yolov5 is a great algorithm, but I'm having some problems. Specifically, I want to add some private information to the .pt file, can this be done?

### Additional

_No response_


what is moasic and what is its default and how to delete it

what is the meaning of moasic

where I can find its default parameter

how to stop moasic and  stop augmentation in general

I use only this line is it augment data by default or not? how to stop augmentation if exist
```
!python train.py --img 640 --batch 16 --epochs 400 --data /mydrive/data.yaml \
  --weights /mydrive/yolov5s.pt --cache --project /mydrive/train/
```


[onnx export.py error] Unsupported ONNX opset version

`ONNX: starting export with onnx 1.10.1...`
`ONNX: export failure: Unsupported ONNX opset version: 13`

I'm using
yolov5-5.0, pytorch1.7.0+cu101 and python3.7.9.

How to solve it?


Experiments on GhostNet

## ❔Question
I am just wondering about the performance when using GhostNet in experimental.py. Could you please share this experiment?

## Additional context


calculate fps on local system

## ❔Question
I have been using the code to do detection from webcam. How can I know what is the speed of detection (fps) in my local system?


large scale dataset use --cache-images flag

## ❔Question
hello ~ , i have dataset with a million images about 450GB and i want to use --cache-images  accelerate training（i have 128GB RAM），can i split the whole dataset into many sub dataset  and training them one by one（like resume training） ？

## Additional context


### Search before asking

- [X] I have searched the YOLOv5 [issues](https://github.com/ultralytics/yolov5/issues) and [discussions](https://github.com/ultralytics/yolov5/discussions) and found no similar questions.

### Question

I am making UI by QT for Yolov5 training. Where is making the result image (results.png) after training?

I would like to draw the graph for (train/box_loss), (metrics/precision), and (metrics/recall) per each an epoch every time an epoch of the train is finished.

 Where is making the result image (results.png) after training?

Thank you for your help.

### Additional

_No response_


Significance of --half?

### Search before asking

- [X] I have searched the YOLOv5 [issues](https://github.com/ultralytics/yolov5/issues) and [discussions](https://github.com/ultralytics/yolov5/discussions) and found no similar questions.

### Question

Can you please let me know the significance of --half during training process....

### Additional

_No response_


Training speed and memory

I noticed your instructions about training,
Run commands below to reproduce results on COCO dataset (dataset auto-downloads on first use). Training times for YOLOv5s/m/l/x are 2/4/6/8 days on a single V100 (multi-GPU times faster). Use the largest --batch-size your GPU allows (batch sizes shown for 16 GB devices).
I want to train from scratch on the coco dataset.(A100 x1).The code was just downloaded.

The following is the situation during my training.The specific parameters can be seen in the screenshot.
python train.py --cfg models/yolov5s.yaml --data data/coco.yaml --device 0 --batch-size 64    -> 16min/epoch
python train.py --cfg models/yolov5s.yaml --data data/coco.yaml --device 0 --batch-size 128  ->16min/epoch
python train.py --cfg models/yolov5s.yaml --data data/coco.yaml --device 0 --batch-size 192  ->20min/epoch
python train.py --cfg models/yolov5s.yaml --data data/coco.yaml --device 0 --batch-size 192 --workers 16->16min/epoch
![sendpix7](https://user-images.githubusercontent.com/39581901/127759471-a110c68f-d1d4-4580-afd2-ae8c8a17ef4a.jpg)
My question
1. Why I increased the batch size but the time required for training did not decrease
2. The relationship between workers and batch size, because I noticed that you seem to set it to a maximum of 8 in the code (why it is 8),
3. When epoch=0 and 1, the GPU memory has changed, about x1.5? What may be the reason for this,


Can we return objectness score and class score?

## ❔Question
I am wondering if it is possible to return confidence scores for objectness and classification separately for each predicted box during inference? I might be conceptually off base here, but I am interested in understanding if the model is unsure if the box itself is correct or if the class it is assigning to the box is correct.  My understanding is the `conf` that is returned now is a combo of the two?


For online cutting training and detection can be improve

## 🚀 Feature

For big image training, usually people thinking about to cut the images, but yolov5 can only resize the image to small size. Such as VisDrone dataset, the smallest image can have 960*540 size, if resize to 640*640, size would be 640*360, but the target in dataset mostly are small object, resize the image make the target become more smaller, but if use bigger resolution, the cuda memory would exceed.

So I thought online cutting training and detection would be a good feature for yolov5 to improve, although cutting image would also increase the train time, but it would be a great idea for people who don't have large computing power GPU, also I think cutting image would be effective for small object detection. Although it's not a new idea in detection, it would be a useful way for people to their own detector.


img-weights

## ❔Question
parser.add_argument('--img-weights', action='store_true', help='use weighted image selection for training')
in order to make --iimg-weights work,  what else I need to do?
dataset = LoadImagesAndLabels(path, imgsz, batch_size,
                                    augment=augment,  # augment images
                                    hyp=hyp,  # augmentation hyperparameters
                                    rect=rect,  # rectangular training
                                    cache_images=cache,
                                    single_cls=opt.single_cls,
                                    stride=int(stride),
                                    pad=pad),
 should I add an extra param image_weights=True??

## Additional context


why can't I reproduce the mAP provided by README.md（v6.1）？

### Search before asking

- [X] I have searched the YOLOv5 [issues](https://github.com/ultralytics/yolov5/issues) and [discussions](https://github.com/ultralytics/yolov5/discussions) and found no similar questions.

### Question

I used the method recommended by README.md(v6.1) to reproduce the mAP, but I failed.
'python train.py --data coco.yaml --cfg yolov5s.yaml --weights ' ' --hyp hyp.scratch-low.yaml --img 640 --batch-size 64 --epochs 300' .
All is default value,then I got the best mAP（yolov5s） is 37.057%(the best mAP verified at the end of each epoch, 5000 images), it still has a gap of 0.4% mAP(37.4%).
Similarly, I reproduced the mAP（yolov5n），27.586%----28.0%，Never get published results.
My GPU is GTX NVIDIA RTX A4000（16116MiB）, and I think it may be enough.

Is this a normal error caused by equipment（GPU) differences, or are there other reasons？

### Additional

_No response_


Is YOLOv5 able to detect a specific number of classes according to the project's need, like just 2 or 3 classes?

### Search before asking

- [X] I have searched the YOLOv5 [issues](https://github.com/ultralytics/yolov5/issues) and [discussions](https://github.com/ultralytics/yolov5/discussions) and found no similar questions.

### Question

Hi, I'm using YOLOv5 in my project and I have a question. If I use "--classes " it could detect one type of class, but is there anyway that I can detect more than one type, like 2 or 3 different types? I've already tried "-- classes 0 1" or "-- classes [0] [1]" but without success. Thanks for the help!

### Additional

_No response_


Average Precision for each class

## Is there any way to see the average precision for each class?

I have run my model for 1,000 epochs and I have a bunch of metrics (which are AMAZING by the way, thanks so making it so easy to see them!) and I have mAP, but I was wondering if there was a way to see the AP for each class? Like a table or something.

In addition, is it possible to see the precision-recall graphs for each class? I can see something in the images tab on wandb, but as I have 80 classes, it looks very messy.


Format of data for testing trained model

In what format do I need to feed the validation dataset to the val.py file? Should images and markup be in the same folder or in different ones? In what format should the coordinates of the bounding boxes be in - yolo or something else?


decoding error in preprocessing synthesizer

I get the following error while running `synthesizer_preprocess_audio.py`.

```
Arguments:
    datasets_root:   /home/amin/voice_cloning/libri_100
    out_dir:         /home/amin/voice_cloning/libri_100/SV2TTS/synthesizer
    n_processes:     None
    skip_existing:   True
    hparams:

Using data from:
    /home/amin/voice_cloning/libri_100/LibriSpeech/train-clean-100
LibriSpeech:   0%|                                                                                | 0/502 [00:00<?, ?speakers/s]
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/amin/voice_cloning/Real-Time-Voice-Cloning-master/synthesizer/preprocess.py", line 62, in preprocess_speaker
    alignments = [line.rstrip().split(" ") for line in alignments_file]
  File "/home/amin/voice_cloning/Real-Time-Voice-Cloning-master/synthesizer/preprocess.py", line 62, in <listcomp>
    alignments = [line.rstrip().split(" ") for line in alignments_file]
  File "/usr/lib/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa2 in position 37: invalid start byte
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "synthesizer_preprocess_audio.py", line 52, in <module>
    preprocess_librispeech(**vars(args))
  File "/home/amin/voice_cloning/Real-Time-Voice-Cloning-master/synthesizer/preprocess.py", line 36, in preprocess_librispeech
    for speaker_metadata in tqdm(job, "LibriSpeech", len(speaker_dirs), unit="speakers"):
  File "/home/amin/.local/lib/python3.6/site-packages/tqdm/std.py", line 1130, in __iter__
    for obj in iterable:
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 735, in next
    raise value
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa2 in position 37: invalid start byte
```

Can anyone help? It can save a lot of time for me.
Thanks.


Error in macOS when trying to launch the toolbox

Traceback (most recent call last):
  File "/Users/luke/Documents/Real-Time-Voice-Cloning-master/demo_toolbox.py", line 2, in <module>
    from toolbox import Toolbox
  File "/Users/luke/Documents/Real-Time-Voice-Cloning-master/toolbox/__init__.py", line 1, in <module>
    from toolbox.ui import UI
  File "/Users/luke/Documents/Real-Time-Voice-Cloning-master/toolbox/ui.py", line 6, in <module>
    from encoder.inference import plot_embedding_as_heatmap
ModuleNotFoundError: No module named 'encoder.inference'


missing SV2TTS/

Hey, I'm trying to finetune the pretrained model but it looks like I am missing the SV2TTS/ directory which contains train.txt, etc.
I have a saved_models/ directory which has three *.pt files for the three components of this TTS architecture.


Import Error

Hey, i am trying to run this code and everytime i run demo_toolbox.py there comes an error "failed to load qt binding" i tried reinstalling matplotlib and also tried installing PYQt5 .

Need Help !!!


ModuleNotFoundError: No module named 'tensorflow.contrib.seq2seq'

When running demo_cli.py

Python = 3.7.4
TensorFlow = 2.0 RC
CUDA = 10.1
cuDNN = Installed for right CUDA version
Windows = 10


Speaker verification implementation

I need just the speaker verification part which is the implementation of [GENERALIZED END-TO-END LOSS FOR SPEAKER VERIFICATION](https://arxiv.org/pdf/1710.10467.pdf) paper, how I can proceed to get it please?


Output audio spectrum - low frequences

Hi, Im am trying to train new model in polish language but after 476k steps output sound is very "robotic". I was trying to find why that's happened and noticed (based on my output and @blue-fish samples: https://blue-fish.github.io/experiments/RTVC-FT-1.html) that spectrum of this model don't include high frequences compared to google. Both in logarithmic scale.

Our output:

Google: (take a note its logarithmic scale)

Do you have any idea how to improve this?


Requirements.txt failed to install with obscure issue with installing audioread

I ran into a few issues along the way that I was able to solve, namely errors like this:

      WARNING: Failed to write executable - trying to use .deleteme logic
    ERROR: Could not install packages due to an OSError: [WinError 2] The system cannot find the file specified:
    'C:\\Python310\\Scripts\\f2py.exe' -> 'C:\\Python310\\Scripts\\f2py.exe.deleteme'

I fixed these by adding `--user` to the pip command.

I also had to change requirements.txt to a newer version of numpy (1.22.1) to prevent it from failing to install due to older versions not being compatible with the version of Python I already have installed (3.10.6)

But now I'm stuck on this one:

    Requirement already satisfied: jsonpointer>=1.9 in c:\users\michael\appdata\roaming\python\python310\site-packages (from jsonpatch->visdom==0.1.8.9->-r R:\requirements.txt (line 15)) (2.3)
    Using legacy 'setup.py install' for umap-learn, since package 'wheel' is not installed.
    Using legacy 'setup.py install' for visdom, since package 'wheel' is not installed.
    Using legacy 'setup.py install' for audioread, since package 'wheel' is not installed.
    Using legacy 'setup.py install' for pynndescent, since package 'wheel' is not installed.
    Installing collected packages: audioread, visdom, SoundFile, sounddevice, scikit-learn, resampy, pooch, matplotlib, pynndescent, librosa, umap-learn
      Running setup.py install for audioread ... error
      error: subprocess-exited-with-error

      × Running setup.py install for audioread did not run successfully.
      │ exit code: 1
      ╰─> [40 lines of output]
          C:\Users\michael\AppData\Local\Temp\pip-install-nat_itg2\audioread_fa5fbfcd88364fc89c7b2a9e454b5549\setup.py:17: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
            import imp
          running install
          C:\Users\michael\AppData\Roaming\Python\Python310\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
            warnings.warn(
          Traceback (most recent call last):
            File "C:\Users\michael\AppData\Roaming\Python\Python310\site-packages\setuptools\_distutils\util.py", line 258, in subst_vars
              return _subst_compat(s).format_map(lookup)
          KeyError: 'py_version_nodot_plat'

          During handling of the above exception, another exception occurred:

          Traceback (most recent call last):
            File "<string>", line 2, in <module>
            File "<pip-setuptools-caller>", line 34, in <module>
            File "C:\Users\michael\AppData\Local\Temp\pip-install-nat_itg2\audioread_fa5fbfcd88364fc89c7b2a9e454b5549\setup.py", line 27, in <module>
              setup(name='audioread',
            File "C:\Users\michael\AppData\Roaming\Python\Python310\site-packages\setuptools\__init__.py", line 153, in setup
              return distutils.core.setup(**attrs)
            File "C:\Users\michael\AppData\Roaming\Python\Python310\site-packages\setuptools\_distutils\core.py", line 148, in setup
              return run_commands(dist)
            File "C:\Users\michael\AppData\Roaming\Python\Python310\site-packages\setuptools\_distutils\core.py", line 163, in run_commands
              dist.run_commands()
            File "C:\Users\michael\AppData\Roaming\Python\Python310\site-packages\setuptools\_distutils\dist.py", line 967, in run_commands
              self.run_command(cmd)
            File "C:\Users\michael\AppData\Roaming\Python\Python310\site-packages\setuptools\_distutils\dist.py", line 985, in run_command
              cmd_obj.ensure_finalized()
            File "C:\Users\michael\AppData\Roaming\Python\Python310\site-packages\setuptools\_distutils\cmd.py", line 107, in ensure_finalized
              self.finalize_options()
            File "C:\Users\michael\AppData\Roaming\Python\Python310\site-packages\setuptools\command\install.py", line 45, in finalize_options
              orig.install.finalize_options(self)
            File "C:\Users\michael\AppData\Roaming\Python\Python310\site-packages\setuptools\_distutils\command\install.py", line 381, in finalize_options
              self.expand_dirs()
            File "C:\Users\michael\AppData\Roaming\Python\Python310\site-packages\setuptools\_distutils\command\install.py", line 563, in expand_dirs
              self._expand_attrs(['install_purelib', 'install_platlib',
            File "C:\Users\michael\AppData\Roaming\Python\Python310\site-packages\setuptools\_distutils\command\install.py", line 553, in _expand_attrs
              val = subst_vars(val, self.config_vars)
            File "C:\Users\michael\AppData\Roaming\Python\Python310\site-packages\setuptools\_distutils\util.py", line 260, in subst_vars
              raise ValueError(f"invalid variable {var}")
          ValueError: invalid variable 'py_version_nodot_plat'
          [end of output]

      note: This error originates from a subprocess, and is likely not a problem with pip.
    error: legacy-install-failure

    × Encountered error while trying to install package.
    ╰─> audioread

    note: This is an issue with the package mentioned above, not pip.
    hint: See above for output from the failure.

I'm not sure if the issue is due to "setup.py install" being deprecated; if that's the case I have no idea what the fix is because I think this is being required somewhere else - maybe another package needs a newer version? But I have no idea which one.

I also thought maybe it could be that wheel wasn't installed, `since package 'wheel' is not installed.` but when I try to install it, it says it's already installed:

    C:\> pip install wheel --user

    Requirement already satisfied: wheel in c:\python310\lib\site-packages (0.37.1)

There's also the invalid variable error, but I have no idea what this is talking about.


I can't use Dataset/Speaker/Utterance

I can't use the upper section in the software. when loading it shows:
Warning: you did not pass a root directory for datasets as argument.
How can I fix this?
Thank you


problem with utils.argutils  in python 3.6

Hi under win 10 64 bits trying using python 3.6 it failed to import the print_args wiht the fact that he can't find the argutils.
think i have a relative import error but can't solve it

btw nice job on what i heard on the youtube demo
if i mnaully try to import the utils from the root dir seems he load another utils files


Using a different speaker encoder

Hello, I really appreciate the work on display here. I was just wondering if I could use a different speaker encoder. If someone used a different encoder, could you explain the difficulties of replacing the encoder and how the results were different from the speaker encoder already in use?


preprocessing VoxCele2 is not working

While running encoder_preprocess on voxceleb2 dataset, I'm getting the following warning and nothing else happens. Not sure why?

```
raw: Preprocessing data for 5994 speakers.
raw:   0%|                                                                                | 0/5994 [00:00<?, ?speakers/s]
/home/amin/.local/lib/python3.6/site-packages/librosa/core/audio.py:161: UserWarning: PySoundFile failed. Trying audioread instead.
  warnings.warn('PySoundFile failed. Trying audioread instead.')
/home/amin/.local/lib/python3.6/site-packages/librosa/core/audio.py:161: UserWarning: PySoundFile failed. Trying audioread instead.
  warnings.warn('PySoundFile failed. Trying audioread instead.')
```


vocoder_dataset.py ValueError

I am trying to use the Librispeech dataset to train the vocoder.
And I got a ValueError while training.
```numpy.random._bounded_integers._rand_int32 ValueError: low >= high```

It occurs in line 61 of vocoder_dataset.py,
```mel_offsets = [np.random.randint(0, offset) for offset in max_offsets]```
So I assume there is something wrong with the value of offset? e.g. offset=0 so np.random.randint could not generate a number [0, 0)?
Did anyone encountered this problem too?