[TikTok] New API fix adds non-playable video codec in available formats

### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting that yt-dlp is broken on a **supported** site
- [X] I've verified that I have **updated yt-dlp to nightly or master** ([update instructions](https://github.com/yt-dlp/yt-dlp#update-channels))
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

Global

### Provide a description that is worded well enough to be understood

Hi! New API fix adds for some videos a new codec bytevc2 which cannot be played by multimedia players (I used VLC). By default yt_dlp chooses normal codec, but I use `-S res:1080,vcodec:avc1,ext:mp4:m4a` format selection, so yt_dlp downloads video with this new codec.

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [ ] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
user@host:~$ python3 -m yt_dlp -F https://vm.tiktok.com/ZMMPDNEJL/
[vm.tiktok] Extracting URL: https://vm.tiktok.com/ZMMPDNEJL/
[vm.tiktok] ZMMPDNEJL: Downloading webpage
[TikTok] Extracting URL: https://www.tiktok.com/@soyko_max/video/7351538939712359713?_t=8l5sekUiLWo&_r=1
[TikTok] 7351538939712359713: Downloading video feed
[info] Available formats for 7351538939712359713:
ID                    EXT RESOLUTION │   FILESIZE  TBR PROTO │ VCODEC  ACODEC MORE INFO
───────────────────────────────────────────────────────────────────────────────────────────────────────────────
download_addr-0       mp4 720x1280   │    1.31MiB      https │ h264    aac    Download video, watermarked (API)
download_addr-1       mp4 720x1280   │    1.31MiB      https │ h264    aac    Download video, watermarked
download_addr-2       mp4 720x1280   │    1.31MiB      https │ h264    aac    Download video, watermarked
h264_540p_986746-0    mp4 1048x576   │    1.24MiB 986k https │ h264    aac    Direct video (API)
h264_540p_986746-1    mp4 1048x576   │    1.24MiB 986k https │ h264    aac    Direct video
h264_540p_986746-2    mp4 1048x576   │    1.24MiB 986k https │ h264    aac    Direct video
bytevc1_540p_263555-0 mp4 1048x576   │  339.96KiB 263k https │ h265    aac    Playback video (API)
bytevc1_540p_263555-1 mp4 1048x576   │  339.96KiB 263k https │ h265    aac    Playback video
bytevc1_540p_263555-2 mp4 1048x576   │  339.96KiB 263k https │ h265    aac    Playback video
bytevc1_540p_344910-0 mp4 1048x576   │  444.91KiB 344k https │ h265    aac    Playback video (API)
bytevc1_540p_344910-1 mp4 1048x576   │  444.91KiB 344k https │ h265    aac    Playback video
bytevc1_540p_344910-2 mp4 1048x576   │  444.91KiB 344k https │ h265    aac    Playback video
bytevc1_540p_507345-0 mp4 1048x576   │  654.43KiB 507k https │ h265    aac    Direct video (API)
bytevc1_540p_507345-1 mp4 1048x576   │  654.43KiB 507k https │ h265    aac    Direct video
bytevc1_540p_507345-2 mp4 1048x576   │  654.43KiB 507k https │ h265    aac    Direct video
bytevc2_720p_616180-0 mp4 1280x704   │  794.82KiB 616k https │ bytevc2 aac    Playback video (API)
bytevc2_720p_616180-1 mp4 1280x704   │  794.82KiB 616k https │ bytevc2 aac    Playback video
bytevc2_720p_616180-2 mp4 1280x704   │  794.82KiB 616k https │ bytevc2 aac    Playback video
bytevc1_720p_595186-0 mp4 1280x704   │  767.74KiB 595k https │ h265    aac    Playback video (API)
bytevc1_720p_595186-1 mp4 1280x704   │  767.74KiB 595k https │ h265    aac    Playback video
bytevc1_720p_595186-2 mp4 1280x704   │  767.74KiB 595k https │ h265    aac    Playback video

user@host:~$ python3 -m yt_dlp -vU -S res:1080,vcodec:avc1,ext:mp4:m4a https://vm.tiktok.com/ZMMPDNEJL/
[debug] Command-line config: ['-vU', '-S', 'res:1080,vcodec:avc1,ext:mp4:m4a', 'https://vm.tiktok.com/ZMMPDNEJL/']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2024.03.10 from yt-dlp/yt-dlp [615a84447]
[debug] Lazy loading extractors is disabled
[debug] Python 3.10.12 (CPython x86_64 64bit) - Linux-5.15.0-101-generic-x86_64-with-glibc2.35 (OpenSSL 3.0.2 15 Mar 2022, glibc 2.35)
[debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2
[debug] Optional libraries: Cryptodome-3.20.0, brotli-1.1.0, certifi-2024.02.02, mutagen-1.47.0, requests-2.31.0, secretstorage-3.3.1, sqlite3-3.37.2, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1807 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: stable@2024.03.10 from yt-dlp/yt-dlp
yt-dlp is up to date (stable@2024.03.10 from yt-dlp/yt-dlp)
[vm.tiktok] Extracting URL: https://vm.tiktok.com/ZMMPDNEJL/
[vm.tiktok] ZMMPDNEJL: Downloading webpage
[TikTok] Extracting URL: https://www.tiktok.com/@soyko_max/video/7351538939712359713?_t=8l5sekUiLWo&_r=1
[debug] [TikTok] iid=7351149742343391009
[TikTok] 7351538939712359713: Downloading video feed
[debug] Sort order given by user: res:1080, vcodec:avc1, ext:mp4:m4a
[debug] Sort order given by extractor: quality, codec, size, br
[debug] Formats sorted by: hasvid, ie_pref, res:1080(1080.0), vcodec:avc1(7), vext:mp4(6), aext:m4a(8), quality, acodec, size, br, lang, fps, hdr:12(7), channels, asr, proto, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] 7351538939712359713: Downloading 1 format(s): bytevc2_720p_616180-2
[debug] Invoking http downloader on "https://v16m.byteicdn.com/366e989edb75b43b4e545b0f6f94180e/6607a32c/video/tos/useast2a/tos-useast2a-ve-0068-euttp/oIEDMCMe2rmeIExnh6mGk70AEkerG1aIRgfFj6/?a=0&bti=OHYpOTY0Zik3OjlmOm01MzE6ZDQ0MDo%3D&ch=0&cr=13&dr=0&lr=all&cd=0%7C0%7C0%7C&cv=1&br=1202&bt=601&cs=5&ds=3&ft=teSL~8QLodzR12NvvEh3hIxR34DaRq_45SY&mime_type=video_mp4&qs=14&rc=aTwzOGdkOGRnMzw1NDQ1OUBpam9pbng5cjx4cjMzZjczM0AvL18yYmBgXjMxXy4tLy5eYSNsa2FnMmRrZy9gLS1kMWNzcw%3D%3D&vvpl=1&l=20240329232904ECF8D33243F1CE142F0B&btag=e00088000&cc=10"
[download] Destination: Оригинал) [7351538939712359713].mp4
[download] 100% of  794.82KiB in 00:00:00 at 7.49MiB/s
```


Disable g4f logging completely

**Bug description**
In my script I have my customized logging, but whenever I use it it prints 2 times (one from my logger, one from g4f logger).
How can I turn off the logger inside the library? Already tried a bunch of stuff with no results.

P.S. Are you using the root logger maybe? If that is the case, please use it with the module name

ex.
1. Create a new logger in new class
2. Set logging level to DEBUG
3. Log something
4. Enjoy duplicated output

**Screenshots**
![image](https://github.com/user-attachments/assets/c24a1b65-b513-4449-8842-870552289de0)

**Environment**
- python version: 3.11
- location ( are you in a cloudfare flagged country ) ? nope


BUG: loc should not fallback for integer indexing for multi-index


DataFrame.ix losing row ordering when index has duplicates

``` python
import pandas as pd

ind = ['A', 'A', 'B', 'C']i
df = pd.DataFrame({'test':range(len(ind))}, index=ind)

rows = ['C', 'B']
res = df.ix[rows]
assert rows == list(res.index) # fails
```

The problem is that the resulting DataFrame keeps the ordering of the `df.index` and not the `rows` key. You'll notice that the `rows` key doesn't reference a duplicate value.


good first issue
frontend
severity:low
small effort

Rename and/or properly document the two different `changeAgentState` functions

There are two `changeAgentState` functions that should probably be renamed and properly documented to avoid confusion for the future.

https://github.com/OpenDevin/OpenDevin/blob/01ce1e35b5b40e57d96b15a7fc9bee4eb8f6966d/frontend/src/state/agentSlice.tsx#L10-L12

https://github.com/OpenDevin/OpenDevin/blob/01ce1e35b5b40e57d96b15a7fc9bee4eb8f6966d/frontend/src/services/agentStateService.ts#L7-L18


Align rotation of input faces for GAN conversions

Currently, the extractor finds a rotation matrix for each face using umeyama so it can generate a faceset with all the faces mostly upright. Unfortunately this rotation matrix isn't stored in the alignments file, only the bbox (of the un-rotated face) and facial alignments. For the GAN model, when it comes time to convert, the faces aren't rotated upright before being fed through the model so I doubt anyone has been able to get good results for faces that aren't completely upright.

I propose we store the rotation matrix in the alignments file during extract, then at conversion, re-apply it to the cropped face to make it upright before feeding through the model. The swapped output face then needs to be rotated in the inverse direction to match it with the frame again. Hopefully this is possible.


item_dropped signal should pass response arg as item_scraped does

I highly use request and response.meta in item_scraped signal handler.
Why item_dropped doesn't pass response argument as well as item_scraper does?


Hello,
I would like to suggest implementing my paper: Differential Diffusion: Giving Each Pixel Its Strength.
The paper allows a user to edit a picture by a change map that describes how much each region should change.
The editing process is typically guided by textual instructions, although it can also be applied without guidance.
We support both continuous and discrete editing.
Our framework is training and fine tuning free! And has negligible penalty of the inference time.
Our implementation is diffusers-based.
We already tested it on 4 different diffusion models (Kadinsky, DeepFloyd IF, SD, SD XL).
We are confident that the framework can also be ported to other diffusion models, such as SD Turbo, Stable Cascade, and amused.
I notice that you usually stick to white==change convention, which is opposite to the convention we used in the paper.
The paper can be thought of as a generalization to some of the existing techniques.
A black map is just regular txt2img ("0"),
A map of one color (which isn't black) can be thought as img2img,
A map of two colors which one color is white can be thought as inpaint.
And the rest? It's completely new!
In the paper, we suggest some further applications such as soft inpainting and strength visualization.

Site:
https://differential-diffusion.github.io/
Paper:
https://differential-diffusion.github.io/paper.pdf
Repo:
https://github.com/exx8/differential-diffusion


ddof for np.std in df.agg changes depending on how given & lambda expression does not work correctly in a list of functions

#### Code Sample, a copy-pastable example if possible

```python
In [31]: import numpy as np

In [32]: import pandas as pd

In [33]: df = pd.DataFrame(np.arange(6).reshape(3, 2), columns=['A', 'B'])

In [34]: df
Out[34]:
   A  B
0  0  1
1  2  3
2  4  5

In [35]: df.agg(np.std)  # Behavior of ddof=0
Out[35]:
A    1.632993
B    1.632993
dtype: float64

In [36]: df.agg([np.std])  # Behavior of ddof=1
Out[36]:
       A    B
std  2.0  2.0

In [37]: # So how to get the ddof=0 behavior when giving a list of functions?

In [39]: df.agg([lambda x: np.std(x)])  # This gives a numerically unexpected result.
Out[39]:
         A        B
  <lambda> <lambda>
0      0.0      0.0
1      0.0      0.0
2      0.0      0.0

In [40]: df.agg([np.mean, lambda x: np.std(x)])  # This gives an error.
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-40-52f4ec4195b5> in <module>()
----> 1 df.agg([np.mean, lambda x: np.std(x)])

/Users/ikeda/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages/pandas/core/frame.py in aggregate(self, func, axis, *args, **kwargs)
   4740         if axis == 0:
   4741             try:
-> 4742                 result, how = self._aggregate(func, axis=0, *args, **kwargs)
   4743             except TypeError:
   4744                 pass

/Users/ikeda/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages/pandas/core/base.py in _aggregate(self, arg, *args, **kwargs)
    537             return self._aggregate_multiple_funcs(arg,
    538                                                   _level=_level,
--> 539                                                   _axis=_axis), None
    540         else:
    541             result = None

/Users/ikeda/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages/pandas/core/base.py in _aggregate_multiple_funcs(self, arg, _level, _axis)
    594         # if we are empty
    595         if not len(results):
--> 596             raise ValueError("no results")
    597
    598         try:

ValueError: no results

```
#### Problem description

When using, e.g., `df.agg`, the `ddof` (degrees of freedom) value for the function `np.std` changes depending on how the function is given (single function or a list of functions), which may be so confusing for many people. I believe the behavior should be unified in some way.

Furthermore, I could not find the way to obtain to the `np.std` result with `ddof=0` by supplying it as one of the members of a list of functions. The `lambda` expression does not work well in a list of functions (this gives numerically unexpected results or even gives errors). This prohibits us to use many useful methods like `df.agg`, `df.apply`, and `df.describe` when we hope the `ddof=0` behavior.

From https://github.com/pandas-dev/pandas/issues/13344, I guess Developers prefer the `ddof=1` behavior in pandas. So the expected behavior should be as below.

#### Expected Output
```
In [35]: df.agg(np.std)  # Behavior of ddof=1
Out[35]:
A    2.0
B    2.0
dtype: float64

In [38]: df.agg([lambda x: np.std(x)])  # To obtain the ddof=0 results
Out[38]:
                     A             B
<lambda>      1.632993      1.632993

In [41]: df.agg([np.mean, lambda x: np.std(x)])
                     A             B
mean          2.0           3.0
<lambda>      1.632993      1.632993
```

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None

pandas: 0.21.0
pytest: 3.0.7
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.13.3
scipy: 0.19.0
pyarrow: None
xarray: None
IPython: 5.3.0
sphinx: 1.5.6
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: 1.2.1
tables: 3.3.0
numexpr: 2.6.2
feather: None
matplotlib: 2.0.2
openpyxl: 2.4.7
xlrd: 1.0.0
xlwt: 1.2.0
xlsxwriter: 0.9.6
lxml: 3.7.3
bs4: 4.6.0
html5lib: 0.999
sqlalchemy: 1.1.9
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>


spin docs --no-plot runs the examples

Seen at the EuroScipy sprint

Commands run by spin:
```
$ export SPHINXOPTS=-W -D plot_gallery=0 -j auto
$ cd doc
$ make html
```

Looks like our Makefile does not use SPHINXOPTS the same way as expected:
Probably we have a slightly different way of building the doc

```
❯ make html-noplot -n
sphinx-build -D plot_gallery=0 -b html -d _build/doctrees  -T  . -jauto \
    _build/html/stable
echo
echo "Build finished. The HTML pages are in _build/html/stable."
```


Add support for ONNX-TensorRT conversion for GPT-J6B (and possible bug in rotary embedding)

### Who can help
@patil-suraj

## Information

Model I am using: GPT-J

The problem arises when using:
* [x] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

## Description
I opened this issue for two reasons:
1. This is not strictly a bug report, rather a change that enables converting this model to ONNX and then parsing it using the current TensorRT ONNX parser.
2. Possible implementation bug in GPT-J.

## Details
1. When exporting GPT-J to ONNX using the latest version (v4.16.2), one of the ops that is exported is [SplitToSequence](https://github.com/onnx/onnx/blob/main/docs/Operators.md#SplitToSequence) (along with more Sequence* ops) that is currently not supported in the [TensorRT ONNX parser](https://github.com/onnx/onnx-tensorrt/blob/master/docs/operators.md).
This is entirely due to just 1 line of code that uses `torch.repeat_interleave`. ([relevant line](https://github.com/huggingface/transformers/blob/52d2e6f6e904ef9b75c78716ce77b98196ed837a/src/transformers/models/gptj/modeling_gptj.py#L67))
```
sin, cos = map(lambda t: t[None, offset : x.shape[1] + offset, None, :].repeat_interleave(2, 3), sincos)
```
By replacing `lambda t` with this:
```
lambda t: t.view(-1, 1).repeat(1, 2).view(seq_len, -1)[None, offset : x.shape[1] + offset, None, :]
```
we get the exact same output tensors but now exporting to ONNX doesn't include any Sequence* ops, and TensorRT can parse it successfully.
The suggested function is even faster, although probably not critical in this huge model (benched only on CPU):
```
original: 106 µs ± 20.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
suggested: 32.4 µs ± 6.55 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

2. I was following the implementation in EleutherAI for rotary positional embeddings and I'm trying to understand if this is a bug or I'm simply missing something (would love an explanation if you can spare the time) but there (EleutherAI) they implement this function (rotary positional embedding) using `torch.cat` instead of `torch.repeat_interleave`, as can be seen [here](https://github.com/EleutherAI/gpt-neox/blob/b30afd1d0a1d06220be9b5f2c9c9c1523defba96/megatron/model/positional_embeddings.py#L41).

If I'm not missing something, the EleutherAI version transforms a tensor from
```
[[1,2,3],
 [4,5,6]]
```
to
```
[[1,2,3,1,2,3],
 [4,5,6,4,5,6]]
```
and HF version (using repeat_interleave):
```
[[1,2,3],
 [4,5,6]]
```
to
```
[[1,1,2,2,3,3],
 [4,4,5,5,6,6]]
```
Can anyone confirm the current implementation is indeed correct? Because otherwise `cat` and `repeat_interleave` are very different, and the rest of the implementation doesn't take it into account.


added image sort tool to faceswap

I added image sort tool to faceswap, which very useful to extract one face from various faces

Example original aligned folder:
![fsviewer_2018-03-08_20-08-06](https://user-images.githubusercontent.com/8076202/37161724-ecbfac92-230c-11e8-8346-5d71c471edc7.png)

Sort it by similarity:
`python.exe faceswap\sorttool.py -i %WORKSPACE%\data_src\aligned -by similarity`
result:
![fsviewer_2018-03-08_20-10-27](https://user-images.githubusercontent.com/8076202/37161776-0b94041a-230d-11e8-9841-b77a562f6120.png)

easy delete faces which you dont need:
![fsviewer_2018-03-08_20-12-52](https://user-images.githubusercontent.com/8076202/37161908-5fae29e0-230d-11e8-88c4-9f89632b05f3.png)

Sort by blur:
`python.exe faceswap\sorttool.py -i %WORKSPACE%\data_src\aligned -by blur`

most sharp 00000.png:
![2018-03-08_20-15-42](https://user-images.githubusercontent.com/8076202/37162142-f37376e4-230d-11e8-8eb7-52892f4fd46c.png)

most blurred 00140.png:
![fsviewer_2018-03-08_20-15-51](https://user-images.githubusercontent.com/8076202/37162187-0bc53f48-230e-11e8-8c23-646c72898704.png)


Bing stopped working

**Bug description**
Yesterday, Bing still worked, but today brings up only:
```
Using Bing provider
0, message='Attempt to decode JSON with unexpected mimetype: text/html; charset=utf-8', url=URL('https://www.bing.com/turing/conversation/create?bundleVersion=1.1381.8')
127.0.0.1 - - [14/Dec/2023 20:22:32] "POST /backend-api/v2/conversation HTTP/1.1" 200 -
```

**Screenshots**
![image](https://github.com/xtekky/gpt4free/assets/11407417/3f562cfb-2596-4f65-ba75-4efec77d0f3e)

**Environment**
- python version: 3.12
- location ( are you in a cloudfare flagged country ) : Ukraine


Create an Extra for Better SSL Support

So right now the SSL connections when you use pyOpenSSL, ndg-httspclient, and pyasn1 are more secure than if you just use the stdlib options. However it's hard to actually remember those three things. It would be cool if requests would add an extra to it's setup.py so that people can install requests with betterssl, something like:

``` python
setup(
    extras_require={
        "betterssl": ["pyOpenSSL", "ndg-httpsclient", "pyasn1"],
    },
)
```

Would make it so people can install requests like `pip install requests[betterssl]` and get all of those dependencies without having to manually track those down. It also means people could depend on `requests[betterssl]` instead of just `requests` in their own setup.py's.

Extra name can of course be bikeshed here :)


[Bug]: Generating using LoRA fails with Runtime Error with `Lora/Networks: use old method` enabled

### Is there an existing issue for this?

- [x] I have searched the existing issues and checked the recent builds/commits

### What happened?

I'm on commit 68f336bd994bed5442ad95bad6b6ad5564a5409a, master HEAD at time of posting.
None of my LORAs seem to be working anymore. Normal prompting works fine, but as soon as I try generating after adding a LORA to my prompt I receive the following:

 `RuntimeError: mat1 and mat2 shapes cannot be multiplied (77x3072 and 768x128)`

I'm not well versed in python nor the inner workings of stable diffusion, so I can't debug this myself effectively.
I don't think providing the LORA file or prompt is necessary, as I can reproduce this with any combination of checkpoints and LORAs which would previously work fine.

### Steps to reproduce the problem

1. Select any SD checkpoint
2. txt2image tab
3. Any combination of prompt and negative prompt, doesnt seem to matter
4. Add a LORA to the prompt, no need to even add the activation token.
5. Any generation settings (for my tests I'm using Euler a, 20 steps, 512x512, CFG 7, no scripts, no hires. fix, no face restore).
6. Generate

### What should have happened?

I would expect the LORA to perform as it did in earlier versions with the same configuration, at the very least, generate an image. I haven't done a bisect, but I tried a commit from a week ago or so and it worked fine there. Every time I pull I delete venv and repositories folders beforehand.

### Version or Commit where the problem happens

68f336bd994bed5442ad95bad6b6ad5564a5409a

### What Python version are you running on ?

Python 3.10.x

### What platforms do you use to access the UI ?

Windows

### What device are you running WebUI on?

Nvidia GPUs (RTX 20 above)

### Cross attention optimization

xformers

### What browsers do you use to access the UI ?

Google Chrome

### Command Line Arguments

```Shell
--xformers --reinstall-xformers --precision full --no-half --skip-torch-cuda-test --opt-split-attention
```

### List of extensions

ddetailer, sd-webui-supermerger, stable-diffusion-webui-dataset-tag-editor, stable-diffusion-webui-wd14-tagger

### Console logs

```Shell
venv "C:\sd\sdwebui\webui\venv\Scripts\Python.exe"
Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]
Version: v1.5.1
Commit hash: 68f336bd994bed5442ad95bad6b6ad5564a5409a
Installing xformers
Collecting xformers==0.0.20
  Using cached xformers-0.0.20-cp310-cp310-win_amd64.whl (97.6 MB)
Installing collected packages: xformers
Successfully installed xformers-0.0.20

[notice] A new release of pip available: 22.3.1 -> 23.2
[notice] To update, run: C:\sd\sdwebui\webui\venv\Scripts\python.exe -m pip install --upgrade pip

Launching Web UI with arguments: --xformers --reinstall-xformers --precision full --no-half --skip-torch-cuda-test --opt-split-attention
Check config files...
Done
Loading weights [cb15a7187a] from C:\sd\sdwebui\webui\models\Stable-diffusion\Deliberate-inpainting.safetensors
Creating model from config: C:\sd\sdwebui\webui\configs\v1-inpainting-inference.yaml
LatentInpaintDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.54 M params.
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 84.3s (launcher: 11.6s, import torch: 31.4s, import gradio: 8.7s, setup paths: 10.6s, other imports: 8.2s, opts onchange: 0.4s, setup codeformer: 0.4s, list SD models: 0.3s, load scripts: 11.1s, create ui: 1.0s, gradio launch: 0.5s).
Applying attention optimization: xformers... done.
Model loaded in 10.0s (load weights from disk: 0.8s, create model: 1.2s, apply weights to model: 6.4s, move model to device: 1.5s).
Loading weights [f36b3ca4d1] from C:\sd\sdwebui\webui\models\Stable-diffusion\edgeOfRealism_edgeOfRealismBakedVAE.safetensors
Creating model from config: C:\sd\sdwebui\webui\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Applying attention optimization: xformers... done.
Model loaded in 8.3s (create model: 0.6s, apply weights to model: 5.9s, move model to device: 1.7s).
*** Error completing request
*** Arguments: ('task(yot3zok0bchp1w0)', 'pov of a beautiful asian woman, formal dress, perfect eyes, petite body, in the forest, colorful, yellow leaves, autumn, hair bun, black hair, facing the viewer, bokeh, soft lighting, perfect face, eye contact, brown eyes, <lora:iu_eor_new-000017:1>', 'badhandv4 easynegative ng_deepnegative_v1_75t', [], 20, 0, False, False, 1, 1, 7, -1.0, -1.0, 0, 0, 0, False, 512, 512, False, 0.7, 2, 'Latent', 0, 0, 0, 0, '', '', [], <gradio.routes.Request object at 0x000001E14BCBFBB0>, 0, False, False, 'positive', 'comma', 0, False, False, '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, '', 'None', 30, 4, 0, 0, False, 'None', '<br>', 'None', 30, 4, 0, 0, 4, 0.4, True, 32) {}
    Traceback (most recent call last):
      File "C:\sd\sdwebui\webui\modules\call_queue.py", line 58, in f
        res = list(func(*args, **kwargs))
      File "C:\sd\sdwebui\webui\modules\call_queue.py", line 37, in f
        res = func(*args, **kwargs)
      File "C:\sd\sdwebui\webui\modules\txt2img.py", line 62, in txt2img
        processed = processing.process_images(p)
      File "C:\sd\sdwebui\webui\modules\processing.py", line 677, in process_images
        res = process_images_inner(p)
      File "C:\sd\sdwebui\webui\modules\processing.py", line 783, in process_images_inner
        p.setup_conds()
      File "C:\sd\sdwebui\webui\modules\processing.py", line 1191, in setup_conds
        super().setup_conds()
      File "C:\sd\sdwebui\webui\modules\processing.py", line 364, in setup_conds
        self.uc = self.get_conds_with_caching(prompt_parser.get_learned_conditioning, negative_prompts, self.steps * self.step_multiplier, [self.cached_uc], self.extra_network_data)
      File "C:\sd\sdwebui\webui\modules\processing.py", line 353, in get_conds_with_caching
        cache[1] = function(shared.sd_model, required_prompts, steps)
      File "C:\sd\sdwebui\webui\modules\prompt_parser.py", line 163, in get_learned_conditioning
        conds = model.get_learned_conditioning(texts)
      File "C:\sd\sdwebui\webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 669, in get_learned_conditioning
        c = self.cond_stage_model(c)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\sdwebui\webui\modules\sd_hijack_clip.py", line 234, in forward
        z = self.process_tokens(tokens, multipliers)
      File "C:\sd\sdwebui\webui\modules\sd_hijack_clip.py", line 271, in process_tokens
        z = self.encode_with_transformers(tokens)
      File "C:\sd\sdwebui\webui\modules\sd_hijack_clip.py", line 324, in encode_with_transformers
        outputs = self.wrapped.transformer(input_ids=tokens, output_hidden_states=-opts.CLIP_stop_at_last_layers)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 811, in forward
        return self.text_model(
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 721, in forward
        encoder_outputs = self.encoder(
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 650, in forward
        layer_outputs = encoder_layer(
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 389, in forward
        hidden_states = self.mlp(hidden_states)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 344, in forward
        hidden_states = self.fc1(hidden_states)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\sdwebui\webui\extensions-builtin\Lora\networks.py", line 357, in network_Linear_forward
        return network_forward(self, input, torch.nn.Linear_forward_before_network)
      File "C:\sd\sdwebui\webui\extensions-builtin\Lora\networks.py", line 345, in network_forward
        y = module.forward(y, input)
      File "C:\sd\sdwebui\webui\extensions-builtin\Lora\network_lora.py", line 84, in forward
        return y + self.up_model(self.down_model(x)) * self.multiplier() * self.calc_scale()
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "C:\sd\sdwebui\webui\extensions-builtin\Lora\networks.py", line 357, in network_Linear_forward
        return network_forward(self, input, torch.nn.Linear_forward_before_network)
      File "C:\sd\sdwebui\webui\extensions-builtin\Lora\networks.py", line 337, in network_forward
        y = original_forward(module, input)
      File "C:\sd\sdwebui\webui\venv\lib\site-packages\torch\nn\modules\linear.py", line 114, in forward
        return F.linear(input, self.weight, self.bias)
    RuntimeError: mat1 and mat2 shapes cannot be multiplied (77x3072 and 768x128)
```

### Additional information

None of the extensions listed are used in the context of this issue


MinibatchKMeans bad center reallocation causes duplicate centers

For instance have a look at:

  http://scikit-learn.org/dev/auto_examples/cluster/plot_dict_face_patches.html

some of the centroids are duplicated, presumably because of a bug in the bad cluster reallocation heuristic.


Overriding the MailSender class

I'd like to use the built-in email notification service for when a scraper exceeds a certain memory limit (`MEMUSAGE_NOTIFY_MAIL` setting), but it looks like it's not possible to specify the MailSender class to use to send the email. I don't want to use SMTP, I'd like to use a third-party mail sender (e.g. sendgrid).
Is there a way around this?
Thanks


Impossible to open 'CC:/manim/manim_3_feb/media/videos/example_scenes/480p15/partial_movie_files/SquareToCircle/00000.mp4'

![Screenshot_1](https://user-images.githubusercontent.com/65260808/81756436-479cd380-94ab-11ea-89e1-af386563bfcd.png)
Help me solve this


Allow detecting Dev Drive on Windows

Windows just announced a new [Dev Drive](https://learn.microsoft.com/en-us/windows/dev-drive/) feature, optimised for high I/O scenarios such as build and test. It also works as a very clear signal that the user is a developer and is doing developer-like tasks.

We should add a function to allow querying whether a specific path is on a Dev Drive. The API is relatively low level, and cannot currently be used from Python, but would allow Python apps to detect when the user is operating on a Dev Drive (e.g. installing or compiling something on one), or choose or offer a more performant temporary or cache location than the user directory.

(For a variety of mostly compatibility reasons, there's no way for Windows to redirect `%TEMP%` onto a Dev Drive, but apps that are aware of it can do it for themselves.)

<!-- gh-linked-prs -->
### Linked PRs
* gh-104805
* gh-105054
<!-- /gh-linked-prs -->


Document Reppy Python version support

The optional dependency on reppy for one of the built-in robots.txt parsers is [preventing us from running the extra-dependencies CI job with Python 3.9+](https://github.com/seomoz/reppy/issues/122). https://github.com/seomoz/reppy has not have a commit for ~1.5 years.

So I think we should deprecate the component.

If we don’t, we should document this limitation, and schedule a deprecation for 1 year before Python 3.8 reaches end of life, ~~i.e. in 9 months~~, because once we drop Python 3.8 support we will be forced to remove this component anyway, so giving a deprecation warning 1 year before is probably in the best interest of any user of the component.


Requests doesn't catch requests.packages.urllib3.exceptions.ProxyError and translate it into a requests module specific exception which derives from RequestException as it does for other errors originating from urllib3. This means if trying to catch any exception derived from RequestException so as to treat it specially, the urllib3 ProxyError will be missed.


Outpainting script does not save multiple images when using batch sliders

When using the batch-count slider and the batch-size slider, the outpainting script does not save multiple images, but just the first one.

Looking at the console window we can see the actual processing is happening for all the N images (batch-count * batch-size), but at the end of the process only the first one is saved to disk.


Add numpy doc strings

Add numpy style doc strings to all functions apart from the main.py file.

<details>
<summary>Checklist</summary>

- [X] `gpt_engineer/ai.py`
> • For each function in this file, add or replace the existing docstring with a numpy-style docstring. The docstring should include a brief description of the function, a list of parameters with their types and descriptions, and a description of the return value.

- [X] `gpt_engineer/chat_to_files.py`
> • For each function in this file, add or replace the existing docstring with a numpy-style docstring. The docstring should include a brief description of the function, a list of parameters with their types and descriptions, and a description of the return value.

- [X] `gpt_engineer/collect.py`
> • For each function in this file, add or replace the existing docstring with a numpy-style docstring. The docstring should include a brief description of the function, a list of parameters with their types and descriptions, and a description of the return value.

- [X] `gpt_engineer/db.py`
> • For each function in this file, add or replace the existing docstring with a numpy-style docstring. The docstring should include a brief description of the function, a list of parameters with their types and descriptions, and a description of the return value.

- [X] `gpt_engineer/learning.py`
> • For each function in this file, add or replace the existing docstring with a numpy-style docstring. The docstring should include a brief description of the function, a list of parameters with their types and descriptions, and a description of the return value.

</details>


Allow user to open the preview website in a new window


Need suggestion on contributing TFDPR

# 🌟 New model addition

## Model description
Hi, I would love to try contributing TFDPR . This is the first time to me, so I need some suggestions.
I have followed @sshleifer 's [great PR on TFBart model](https://github.com/huggingface/transformers/commit/829842159efeb1f920cbbb1daf5ad67e0114d0b9) on 4 files :` __init__.py , convert_pytorch_checkpoint_to_tf2.py , utils/dummy_tf_objects.py` and (newly created) `modeling_tf_dpr.py `

Now the TF model works properly and can load Pytorch's weights successfully the same output as Pytorch's counterparts **except** small random noise (1e-5) which I suspect of some dtypes different , but I could not find the cause.

I guess I need to add document on  docs/source/model_doc/dpr.rst , and that's all ?
**My question is do I need to change / fix any other files ? and/or do I need to do some other thing before making PR ?**

<!-- Important information -->
To resolve TF vs. Pytorch naming issues, there's one change regarding `TFBertModel` vs. `TFBertMainLayer` as [discussed here](https://discuss.huggingface.co/t/solved-issue-on-translating-dpr-to-tfdpr-on-loading-pytorch-weights-to-tf-model/1764) .
Thanks to @sshleifer for his help to solve the issue.

## Open source status

* [X] the model implementation is available: (give details)
You can see all the modified codes with test run at :
https://colab.research.google.com/drive/1lU4fx7zkr-Y3CXa3wmHIY8yJhKdiN3DI?usp=sharing
(to easily navigate the changes, please “find on page” for e.g. `TFDPRContextEncoder` )

* [X] the model weights are available: (give details)
At the moment, I use existing Pytorch weights, but will upload TF weights too.

* [X] who are the authors: (mention them, if possible by @gh-username)
@ratthachat


[mask2former] torch.export error for Mask2Former

### System Info

- `transformers` version: 4.46.0.dev0
- Platform: Linux-6.8.0-47-generic-x86_64-with-glibc2.35
- Python version: 3.11.9
- Huggingface_hub version: 0.25.2
- Safetensors version: 0.4.5
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.4.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce RTX 4090

### Who can help?

@amyeroberts, @qubvel, @ylacombe

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
import torch
from transformers import Mask2FormerForUniversalSegmentation

model = Mask2FormerForUniversalSegmentation.from_pretrained(
    "facebook/mask2former-swin-base-coco-panoptic", torchscript=True
)

scripted_model = torch.export.export(model, args=(torch.randn(1, 3, 800, 1280),))
```
which causes
```
UserError: Could not extract specialized integer from data-dependent expression u0 (unhinted: u0).  (Size-like symbols: none)

Potential framework code culprit (scroll up for full backtrace):
  File "/home/philkuz/.pyenv/versions/3.11.9/envs/gml311/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 2132, in run_node
    return node.target(*args, **kwargs)

For more information, run with TORCH_LOGS="dynamic"
For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL="u0"
If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1
For more debugging help, see https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing

User Stack (most recent call last):
  (snipped, see stack below for prefix)
  File "/home/philkuz/dev/transformers/src/transformers/models/mask2former/modeling_mask2former.py", line 2499, in forward
    outputs = self.model(
  File "/home/philkuz/.pyenv/versions/3.11.9/envs/gml311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philkuz/dev/transformers/src/transformers/models/mask2former/modeling_mask2former.py", line 2270, in forward
    pixel_level_module_output = self.pixel_level_module(
  File "/home/philkuz/.pyenv/versions/3.11.9/envs/gml311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philkuz/dev/transformers/src/transformers/models/mask2former/modeling_mask2former.py", line 1395, in forward
    decoder_output = self.decoder(backbone_features, output_hidden_states=output_hidden_states)
  File "/home/philkuz/.pyenv/versions/3.11.9/envs/gml311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philkuz/dev/transformers/src/transformers/models/mask2former/modeling_mask2former.py", line 1319, in forward
    encoder_outputs = self.encoder(
  File "/home/philkuz/.pyenv/versions/3.11.9/envs/gml311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philkuz/dev/transformers/src/transformers/models/mask2former/modeling_mask2former.py", line 1165, in forward
    reference_points = self.get_reference_points(spatial_shapes, valid_ratios, device=inputs_embeds.device)
  File "/home/philkuz/dev/transformers/src/transformers/models/mask2former/modeling_mask2former.py", line 1106, in get_reference_points
    torch.linspace(0.5, height - 0.5, height, dtype=valid_ratios.dtype, device=device),

For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1
For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#constrain-as-size-example

from user code:
   File "/home/philkuz/dev/transformers/src/transformers/models/mask2former/modeling_mask2former.py", line 2499, in forward
    outputs = self.model(
  File "/home/philkuz/.pyenv/versions/3.11.9/envs/gml311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philkuz/dev/transformers/src/transformers/models/mask2former/modeling_mask2former.py", line 2270, in forward
    pixel_level_module_output = self.pixel_level_module(
  File "/home/philkuz/.pyenv/versions/3.11.9/envs/gml311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philkuz/dev/transformers/src/transformers/models/mask2former/modeling_mask2former.py", line 1395, in forward
    decoder_output = self.decoder(backbone_features, output_hidden_states=output_hidden_states)
  File "/home/philkuz/.pyenv/versions/3.11.9/envs/gml311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philkuz/dev/transformers/src/transformers/models/mask2former/modeling_mask2former.py", line 1319, in forward
    encoder_outputs = self.encoder(
  File "/home/philkuz/.pyenv/versions/3.11.9/envs/gml311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/philkuz/dev/transformers/src/transformers/models/mask2former/modeling_mask2former.py", line 1165, in forward
    reference_points = self.get_reference_points(spatial_shapes, valid_ratios, device=inputs_embeds.device)
  File "/home/philkuz/dev/transformers/src/transformers/models/mask2former/modeling_mask2former.py", line 1106, in get_reference_points
    torch.linspace(0.5, height - 0.5, height, dtype=valid_ratios.dtype, device=device),
 ```

### Expected behavior

torch.export works for this model.


scrapy selector fails when large lines are present response

Originally encoutered when scraping [Amazon restaurant](https://www.amazon.com/restaurants/zzzuszimbos0015gammaloc1name-new-york/d/B01HH7CS44?ref_=amzrst_pnr_cp_b_B01HH7CS44_438).
This page contains multiple script tag with lines greater then 64,000 character in one line.
The selector (xpath and css) does not search beyond these lines.

Due to this the following xpath `'//h1[contains(@class, "hw-dp-restaurant-name")]/text()'` to extract name of the restaurant returns empty even though there is a matching tag is present.

PFA the response text at [original_response.html.txt.gz](https://github.com/scrapy/scrapy/files/1631425/original_response.html.txt.gz)


TST/CLN: remove TestData from frame-tests; replace with fixtures

Following review in #22236:
> ok, pls open a new issue that refs this, to remove use of `TestData` in favor of fixtures

Started the process in that PR by creating a `conftest.py` that translates all the current attributes of `TestData` to fixtures, with the following "translation guide":

* `frame` -> `float_frame`
* `frame2` -> `float_frame2`
* `intframe` -> `int_frame`
* `tsframe` -> `datetime_frame`
* `mixed_frame` -> `float_string_frame`
* `mixed_float` -> `mixed_float_frame`
* `mixed_float2` -> `mixed_float_frame2`
* `mixed_int` -> `mixed_int_frame`
* `all_mixed` -> `mixed_type_frame`
* `tzframe` -> `timezone_frame`
* `empty` -> `empty_frame`
* `ts1` -> `datetime_series`
* `ts2` -> `datetime_series_short`
* `simple` -> `simple_frame`

Need to incrementally replace their usages in `pandas/tests/frame/` (example below).

- [x] Create `conftest.py` and translate `TestData`-attributes into fixtures (#22236)
- [x] `test_alter_axes.py` (#22236)
- [x] `test_analytics.py` (#22733)
- [x] `test_api.py` (#22738)
- [x] `test_apply.py` (#22735)
- [x] `test_arithmetic.py` (#22736)
- [x] `test_asof.py` (#25628)
- [x] `test_axis_select_reindex.py` (#25627)
- [x] `test_block_internals.py` (#22926)
- [x] `test_combine_concat.py` (#25634)
- [ ] `test_constructors.py` (#25635)
- [ ] `test_convert_to.py`
- [ ] `test_dtypes.py` (#25636)
- [x] `test_duplicates.py`
- [x] `test_indexing.py` (#25633)
- [x] `test_join.py` (#25639)
- [x] `test_missing.py` (#25640)
- [x] `test_mutate_columns.py` (#25642)
- [ ] `test_nonunique_indexes.py`
- [x] `test_operators.py` (#25641)
- [ ] `test_period.py`
- [ ] `test_quantile.py`
- [ ] `test_query_eval.py`
- [ ] `test_rank.py`
- [ ] `test_replace.py`
- [ ] `test_repr_info.py`
- [ ] `test_reshape.py`
- [ ] `test_sort_values_level_as_str.py`
- [ ] `test_sorting.py`
- [ ] `test_subclass.py`
- [ ] `test_timeseries.py`
- [ ] `test_timezones.py`
- [ ] `test_to_csv.py`
- [ ] `test_validate.py`

Things for follow-ups:
- Remove other class-based test-methods
- Turn tests from class- to function-based

An example from #22236 - before:
```
def test_set_columns(self):
    cols = Index(np.arange(len(self.mixed_frame.columns)))
    self.mixed_frame.columns = cols
    with tm.assert_raises_regex(ValueError, 'Length mismatch'):
        self.mixed_frame.columns = cols[::2]
```
After:
```
def test_set_columns(self, float_string_frame):
    cols = Index(np.arange(len(float_string_frame.columns)))
    float_string_frame.columns = cols
    with tm.assert_raises_regex(ValueError, 'Length mismatch'):
        float_string_frame.columns = cols[::2]
```

Basically, it comes down to replacing all the occurrences of `self.<name>` with `translation_guide[<name>]` (and specifying`<name>` as a parameter to the function).

PS. Note that some fixtures added by #22236 have now been removed by #24885. Please check #24885 which code was removed, in case you should need it for the fixturisation. Alternatively, you can ping me, @jbrockmendel or @jreback.


API Rate Limit Reached with new key

I just create a new key and it's failing to run:
```
Continue (y/n): y
Error:  API Rate Limit Reached. Waiting 10 seconds...
Error:  API Rate Limit Reached. Waiting 10 seconds...
Error:  API Rate Limit Reached. Waiting 10 seconds...
Error:  API Rate Limit Reached. Waiting 10 seconds...
Error:  API Rate Limit Reached. Waiting 10 seconds...
Error:  API Rate Limit Reached. Waiting 10 seconds...
Error:  API Rate Limit Reached. Waiting 10 seconds...
Error:  API Rate Limit Reached. Waiting 10 seconds...
```

![image](https://user-images.githubusercontent.com/1215497/229545231-7b463bc9-4630-45d5-a8cc-41df10e4e4be.png)


Support for ChatGLM

**Description**

[ChatGLM-6B](https://github.com/THUDM/ChatGLM-6B)

A Chinese chat AI based on GLM was released by THU.


StratifiedKFold should do its best to preserve the dataset dependency structure

As highlighted in this [notebook](http://nbviewer.ipython.org/urls/raw.github.com/ogrisel/notebooks/master/Non%2520IID%2520cross-validation.ipynb) the current implementation of `StratifiedKFold` (which is used by default by `cross_val_score` and `GridSearchCV` for classification problems) breaks the dependency structure of the dataset by computing the folds based on the sorted labels.

Instead one should probably do an implementation that performs individual dependency preserving KFold on for each possible label value and aggregate the folds to get the `StratifiedKFold` final folds.

This might incur a refactoring to get rid of the `_BaseKFold` base class. It might also make it easier to implement a `shuffle=True` option for `StratifiedKFold`.


enhancement
good first issue
potential plugin
Stale

Auto-GPT System Awareness

### System Awareness

- [X] I have searched the existing issues

### Summary 💡

Before going out to look at the internet
It would be helpful if upon activation the AI took inventory of the system it was on and shared the available tools and capabilities
and if they were insufficient begin researching and developing GAP tools to use during the session with the expressed request to push the GAP tools via PR back to the community

### Examples 🌈

AI System initializing
- MacOS
- Python3
- Pip
- Shell Commands available...
- Desktop App skills available...

What are your goals?

### Motivation 🔦

usuability


Agent stuck in the "starting task" step--Unsupported Protocol

<!-- You MUST fill out this template. We will close issues that don't include enough information to reproduce -->
#### Describe the bug
<!-- a short description of the problem -->
I asked the agent to build a calculator, but it didn't give me any response, just stuck in the starting step.

#### Setup and configuration
**Current version**:
<!-- run `git log -n 1` to see this -->
```bash
commit e9121b78fed0b5ef36718ca0bf59588c0b094b86 (HEAD -> main)
Author: Xingyao Wang <xingyao6@illinois.edu>
Date:   Sun Apr 7 16:07:59 2024 +0800
```

<!-- tell us everything about your environment -->
**My config.toml and environment vars** (be sure to redact API keys):
```toml
LLM_MODEL="gpt-3.5-turbo-1106"
LLM_API_KEY="already set, and have test in python script, which works"
LLM_EMBEDDING_MODEL="openai"
WORKSPACE_DIR="./workspace"
```

**My model and agent** (you can see these settings in the UI):
* Model: PlannerAgent
* Agent: gpt-3.5-turbo-1106

**Commands I ran to install and run OpenDevin**:
```
make build
make run
```

**Steps to Reproduce**:
run the commands, input: build a calculator with python

**Logs, error messages, and screenshots**:
![image](https://github.com/OpenDevin/OpenDevin/assets/86202027/b078b7b0-446d-4e9d-bc83-ff3c270d9512)
backend:
```
INFO:     127.0.0.1:34564 - "GET /litellm-agents HTTP/1.1" 200 OK
INFO:     127.0.0.1:34572 - "GET /messages/total HTTP/1.1" 200 OK
INFO:     127.0.0.1:34584 - "DELETE /messages HTTP/1.1" 200 OK

==============
STEP 0

PLAN:
build a calculator with python

INFO:
HINT:
You're not currently working on any tasks. Your next action MUST be to mark a task as in_progress.
```

frontend:
```
22:35:39 - opendevin:INFO: sandbox.py:117 - Using workspace directory: /mnt/d/OpenDevin/workspace
22:35:39 - opendevin:INFO: sandbox.py:257 - Container stopped
22:35:39 - opendevin:INFO: sandbox.py:277 - Container started
22:37:54 - opendevin:INFO: sandbox.py:117 - Using workspace directory: /mnt/d/OpenDevin/workspace
```

llm prompt_001:
```
[{'content': '\n# Task\nYou\'re a diligent software engineer AI. You can\'t see, draw, or interact with a\nbrowser, but you can read and write files, and you can run commands, and you can think.\n\nYou\'ve been given the following task:\n\nbuild a calculator with python\n\n## Plan\nAs you complete this task, you\'re building a plan and keeping\ntrack of your progress. Here\'s a JSON representation of your plan:\n\n{\n  "id": "0",\n  "goal": "build a calculator with python",\n  "state": "open",\n  "subtasks": []\n}\n\n\nYou\'re not currently working on any tasks. Your next action MUST be to mark a task as in_progress.\n\nYou\'re responsible for managing this plan and the status of tasks in\nit, by using the `add_task` and `modify_task` actions described below.\n\nIf the History below contradicts the state of any of these tasks, you\nMUST modify the task using the `modify_task` action described below.\n\nBe sure NOT to duplicate any tasks. Do NOT use the `add_task` action for\na task that\'s already represented. Every task must be represented only once.\n\nTasks that are sequential MUST be siblings. They must be added in order\nto their parent task.\n\nIf you mark a task as \'completed\', \'verified\', or \'abandoned\',\nall non-abandoned subtasks will be marked the same way.\nSo before closing a task this way, you MUST not only be sure that it has\nbeen completed successfully--you must ALSO be sure that all its subtasks\nare ready to be marked the same way.\n\nIf, and only if, ALL tasks have already been marked verified,\nyou MUST respond with the `finish` action.\n\n## History\nHere is a recent history of actions you\'ve taken in service of this plan,\nas well as observations you\'ve made. This only includes the MOST RECENT\nten actions--more happened before that.\n\n[]\n\n\nYour most recent action is at the bottom of that history.\n\n## Action\nWhat is your next thought or action? Your response must be in JSON format.\n\nIt must be an object, and it must contain two fields:\n* `action`, which is one of the actions below\n* `args`, which is a map of key-value pairs, specifying the arguments for that action\n\n* `read` - reads the content of a file. Arguments:\n  * `path` - the path of the file to read\n* `write` - writes the content to a file. Arguments:\n  * `path` - the path of the file to write\n  * `content` - the content to write to the file\n* `run` - runs a command on the command line in a Linux shell. Arguments:\n  * `command` - the command to run\n  * `background` - if true, run the command in the background, so that other commands can be run concurrently. Useful for e.g. starting a server. You won\'t be able to see the logs. You don\'t need to end the command with `&`, just set this to true.\n* `kill` - kills a background command\n  * `id` - the ID of the background command to kill\n* `browse` - opens a web page. Arguments:\n  * `url` - the URL to open\n* `think` - make a plan, set a goal, or record your thoughts. Arguments:\n  * `thought` - the thought to record\n* `add_task` - add a task to your plan. Arguments:\n  * `parent` - the ID of the parent task\n  * `goal` - the goal of the task\n  * `subtasks` - a list of subtasks, each of which is a map with a `goal` key.\n* `modify_task` - close a task. Arguments:\n  * `id` - the ID of the task to close\n  * `state` - set to \'in_progress\' to start the task, \'completed\' to finish it, \'verified\' to assert that it was successful, \'abandoned\' to give up on it permanently, or `open` to stop working on it for now.\n* `finish` - if ALL of your tasks and subtasks have been verified or abandoned, and you\'re absolutely certain that you\'ve completed your task and have tested your work, use the finish action to stop working.\n\nYou MUST take time to think in between read, write, run, browse, and recall actions.\nYou should never act twice in a row without thinking. But if your last several\nactions are all `think` actions, you should consider taking a different action.\n\nWhat is your next thought or action? Again, you must reply with JSON, and only with JSON.\n\nYou\'re not currently working on any tasks. Your next action MUST be to mark a task as in_progress.\n', 'role': 'user'}]
```

llm response is empty

#### Additional Context
I also tried to use gpt-4 and got the same result.


* face_recognition version: 1.23
* Python version: 3.6.6
* Operating System: windows 10

### Description
![image](https://user-images.githubusercontent.com/2375460/56864177-e74b0900-69f1-11e9-9cad-d44cc8ca9d3d.png)

### What I Did

```
facerec_from_webcam_multiprocessing.py run Global is not defined. pls fix it, thanks
```


Doc for HDFStore compression unclear on what the default value of None does

The doc for the `HDFStore` class mentions:

```
complevel : int, 0-9, default None
            Specifies a compression level for data.
            A value of 0 disables compression.
```

That doesn't actually answer the question of what compression level is used when the default (None) is used, though. Is None translated further down to 0? it turns out yes, but you have to dig in the code to actually figure that out. And it could as well have been translated eventually to any other value.

Two options:
1. Actually change the default in the `complevel` argument to be "0". (It's an immutable object, so it's fine as a default value for a function argument.)
2. Just adjust the doc in some way.

When the right solution is decided, I can do a pull request with it. Thanks!


git push upstream branch does not exist, wrong command recommended first

<!-- If you have any issue with The Fuck, sorry about that, but we will do what we
can to fix that. Actually, maybe we already have, so first thing to do is to
update The Fuck and see if the bug is still there. -->

<!-- If it is (sorry again), check if the problem has not already been reported and
if not, just open an issue on [GitHub](https://github.com/nvbn/thefuck) with
the following basic information: -->

Recently I noticed a change in a `thefuck` behavior that I use very regularly which I wanted to call out as what I think is an unwanted change. This was introduced very recently, I believe with the 3.31 release. When using `git push` on a git repository where the branch does not exist in the upstream repository, `git` responds with a specific command one should run to create the upstream branch. Prior to version 3.31, `thefuck` seemed to recognize this and made the first suggested Corrected Command was the one `git` recommended. As of version 3.31, `thefuck` instead puts a generic `git push --no-verify` command first, and the one `git` recommended is instead the second result.

In this case where `git` recommends a specific command, `git push --no-verify` doesn't actually help or do what the user wants; you need the `git push --set-upstream origin branch-name` command which `thefuck` now arrives at second. Because of the inconvenience for this particular case, combined with the fact that the first option recommended by `thefuck` isn't functionally valid, the prior behavior is more correct for this particular case.

Below is all the debug information requested in the issue template:

The output of `thefuck --version` (something like `The Fuck 3.1 using Python
3.5.0 and Bash 4.4.12(1)-release`):

    The Fuck 3.31 using Python 3.9.5 and ZSH 5.8

Your system (Debian 7, ArchLinux, Windows, etc.):

    Arch Linux

How to reproduce the bug:

    - In a git repo, create a branch which does not exist in the upstream repository
    - Attempt to push the branch with `git push`
    - You should see an error message saying "fatal: The current branch branch-name has no upstream branch. To push the current branch and set the remote as upstream, use git push --set-upstream origin branch-name"
    - invoke `thefuck`
    - Prior to 3.31, `thefuck` would present as the first option the exact command which git tells you to use (git push --set-upstream origin branch-name).
    - As of 3.31, `thefuck` instead presents as the first option a more generic `git push --no-verify`, and git's recommended command is the second result.

The output of The Fuck with `THEFUCK_DEBUG=true` exported (typically execute `export THEFUCK_DEBUG=true` in your shell before The Fuck):

https://pastebin.com/qpyEcreC

If the bug only appears with a specific application, the output of that application and its version:

    git version 2.32.0

Anything else you think is relevant:

    N/A

<!-- It's only with enough information that we can do something to fix the problem. -->


[Bug] 1.7.1 not support 1.6.0 script

Hello All,

My spider is created by scrapy 1.6.0.
These days, the scrapy updated to 1.7.1, and we found that it cannot support the code build by 1.6.0.

Here is the error:
```
Traceback (most recent call last):
  File "/usr/bin/scrapy", line 6, in <module>
    from scrapy.cmdline import execute
  File "/usr/lib64/python2.7/site-packages/scrapy/cmdline.py", line 10, in <module>
    from scrapy.crawler import CrawlerProcess
  File "/usr/lib64/python2.7/site-packages/scrapy/crawler.py", line 11, in <module>
    from scrapy.core.engine import ExecutionEngine
  File "/usr/lib64/python2.7/site-packages/scrapy/core/engine.py", line 14, in <module>
    from scrapy.core.scraper import Scraper
  File "/usr/lib64/python2.7/site-packages/scrapy/core/scraper.py", line 18, in <module>
    from scrapy.core.spidermw import SpiderMiddlewareManager
  File "/usr/lib64/python2.7/site-packages/scrapy/core/spidermw.py", line 13, in <module>
    from scrapy.utils.conf import build_component_list
  File "/usr/lib64/python2.7/site-packages/scrapy/utils/conf.py", line 4, in <module>
    import configparser
ImportError: No module named configparser
```

Would you please take time to check the issue?

Appreciate for your help in advance.

Thank you.


[Bug]: Browsing is not working

### Is there an existing issue for the same bug?

- [X] I have checked the troubleshooting document at https://github.com/OpenDevin/OpenDevin/blob/main/docs/guides/Troubleshooting.md
- [X] I have checked the existing issues.

### Describe the bug

When I ask a question that requires browsing the web to get the answer, OpenDevin does not use the "browsing" tab.

For instance, I asked
```
Please resolve this pull request: https://github.com/OpenDevin/OpenDevin/issues/1275
```

In trying to resolve the pull request, OpenDevin tried to install Playwright to browse the web instead of using built-in browsing capability, and the browsing tab said "no screenshot available".

### Current Version

```bash
`ghcr.io/opendevin/opendevin:0.3.1`
```

### Installation and Configuration

```bash
export LLM_API_KEY="sk-..."
export WORKSPACE_DIR=$(pwd)/workspace
```

### Model and Agent

_No response_

### Reproduction Steps

1. Ask OpenDevin: `Please resolve this pull request: https://github.com/OpenDevin/OpenDevin/issues/1275`

### Logs, Errors, Screenshots, and Additional Context

It seems that this is a relevant error:

```
STEP 2

23:12:20 - opendevin:INFO: agent_controller.py:89
PLAN
Resolve this pull request: https://github.com/OpenDevin/OpenDevin/issues/1275
23:12:26 - opendevin:INFO: agent_controller.py:107
ACTION
BrowseURLAction(url='https://github.com/OpenDevin/OpenDevin/pull/1275', action=<ActionType.BROWSE: 'browse'>)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
        - Avoid using `tokenizers` before the fork if possible
        - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
23:12:27 - opendevin:INFO: agent_controller.py:160
OBSERVATION
BrowserType.launch: Executable doesn't exist at /root/.cache/ms-playwright/chromium-1112/chrome-linux/chrome
╔════════════════════════════════════════════════════════════╗
║ Looks like Playwright was just installed or updated.       ║
║ Please run the following command to download new browsers: ║
║                                                            ║
║     playwright install                                     ║
║                                                            ║
║ <3 Playwright Team                                         ║
╚════════════════════════════════════════════════════════════╝
```


Nesting FastAPI instances doesn't work very well

Do this:

    main_app = FastAPI()
    sub_api = FastAPI()

    ...
    main_app.router.routes.append(Mount('/subapi', app=sub_api))

`sub_api` will correctly serve ever `/subapi` -- docs, methods, all that. However, the docs will still look for `/openapi.json` (absolute link) when trying to load the openapi spec. Additionally, the spec will not be adjusted to have the correct links, relative to where the module is mounted.

Perhaps this is a corner use case, but a lot of apps might have different collections of routes mounted in different subpaths.


type: bug
status: triage needed
aws:cloudformation
aws:iam

bug: AWS::NoValue produces error when used in IAM policy template

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Current Behavior

When I try to create a role with S3 resource and I use `!Ref AWS::NoValue` for its resource, it fails with errors. It is supposed to be removed from array entry, but it looks like it evaluates as `__aws_no_value__`, which then fails to validate because the value is not in acceptable format for ARN.
(Message: `Resource __aws_no_value__ must be in ARN format or "*".`)

template file `test.template` :
```
AWSTemplateFormatVersion: 2010-09-09

Conditions:
  someCondition: false

Resources:
  SomeRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: SomeRole
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - sts:AssumeRole
      Policies:
        - PolicyName: SomePolicy
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - s3:GetObject
                  - s3:GetObjectVersion
                Resource:
                  - arn:aws:s3:::some-prefix-*/*
                  - !If
                    - someCondition
                    - !Ref arn:aws:s3:::another-prefix-*/*
                    - !Ref AWS::NoValue
```
Executed command:
```
awslocal cloudformation deploy \
  --no-fail-on-empty-changeset \
  --capabilities CAPABILITY_NAMED_IAM \
  --template-file test.template \
  --stack-name "test-stack"
```
Error log produced:
```
2021-08-30T07:38:54:DEBUG:localstack.utils.cloudformation.template_deployer: Error applying changes for CloudFormation stack "test-resources-iam": An error occurred (MalformedPolicyDocument) when calling the PutRolePolicy operation: Resource __aws_no_value__ must be in ARN format or "*". Traceback (most recent call last):
  File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 2083, in _run
    self.do_apply_changes_in_loop(changes, stack, stack_name)
  File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 2154, in do_apply_changes_in_loop
    self.apply_change(change, stack, new_resources, stack_name=stack_name)
  File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 2218, in apply_change
    result = deploy_resource(resource_id, new_resources, stack_name)
  File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 1037, in deploy_resource
    return execute_resource_action(resource_id, resources, stack_name, ACTION_CREATE)
  File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 1152, in execute_resource_action
    resource_id, resources, resource_type, func, stack_name, action_name
  File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 1314, in configure_resource_via_sdk
    run_post_create_actions(action_name, resource_id, resources, resource_type, stack_name, result)
  File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 1414, in run_post_create_actions
    PolicyDocument=doc,
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/opt/code/localstack/.venv/lib/python3.7/site-packages/botocore/client.py", line 705, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.MalformedPolicyDocumentException: An error occurred (MalformedPolicyDocument) when calling the PutRolePolicy operation: Resource __aws_no_value__ must be in ARN format or "*".
```

### Expected Behavior

Create stack without failing

### How are you starting LocalStack?

With the `localstack` script

### Steps To Reproduce

#### How are you starting localstack (e.g., `bin/localstack` command, arguments, or `docker-compose.yml`)

```
FORCE_NONINTERACTIVE=1 \
SERVICES=iam,s3,lambda,cloudformation \
localstack infra start &
```

#### Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

```
awslocal cloudformation deploy \
  --no-fail-on-empty-changeset \
  --capabilities CAPABILITY_NAMED_IAM \
  --template-file test.template \
  --stack-name "test-stack"
```

### Environment

```markdown
- OS:  Ubuntu 20.04
- LocalStack: latest
```

### Anything else?

_No response_


GPT Engineer will not save individual files when given specs that result in many files.

The generated code goes into the logfile however it would be more useful if the tool could make all those files automatically.


Switch to the latest sphinx

The docs fail to build with the current Sphinx (7.2.6):

```
reading sources... [ 48%] topics/downloader-middleware
Extension error (scrapydocs):
Handler <function collect_scrapy_settings_refs at 0x7f81fc663a60> for event 'doctree-read' threw an exception (exception: Next node is not a target)
```

So we should update deps in docs/requirements.txt, fix this (and maybe others) problem and make sure the docs are built correctly.


ReadTimeout when using local LLM

**Bug description**
When hosting the following model; https://huggingface.co/oobabooga/CodeBooga-34B-v0.1 locally using LMStudio 0.2.14 on Linux Mint 21.3 Cinnamon I am sometimes (usually after several iterations when the context gets large) confronted with a ReadTimeout.

MetaGPT main branch, commit id: adb42f4, it reports version: 0.7.4 with pip show metagpt. Used Python 3.9.18.

I used the following code to try out MetaGPT
```
import asyncio
from metagpt.roles.di.data_interpreter import DataInterpreter

async def main(requirement: str = ""):
    di = DataInterpreter()
    await di.run(requirement)

if __name__ == "__main__":
    requirement = "Create a dnd 5th edition graph displaying xp per level based on information from a reputable source determined by Googling. First write results in a CSV and validate the CSV contains multiple records. If the file does not contain records, determine if you can fix the code or whether you need to look at another source. After the CSV files is filled with records, create the graph based on this."

    asyncio.run(main(requirement))
```
I got the below exception
```
Traceback (most recent call last):
  File "metagpt/lib/python3.9/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions
    yield
  File "metagpt/lib/python3.9/site-packages/httpx/_transports/default.py", line 254, in __aiter__
    async for part in self._httpcore_stream:
  File "metagpt/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 367, in __aiter__
    raise exc from None
  File "metagpt/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 363, in __aiter__
    async for part in self._stream:
  File "metagpt/lib/python3.9/site-packages/httpcore/_async/http11.py", line 349, in __aiter__
    raise exc
  File "metagpt/lib/python3.9/site-packages/httpcore/_async/http11.py", line 341, in __aiter__
    async for chunk in self._connection._receive_response_body(**kwargs):
  File "metagpt/lib/python3.9/site-packages/httpcore/_async/http11.py", line 210, in _receive_response_body
    event = await self._receive_event(timeout=timeout)
  File "metagpt/lib/python3.9/site-packages/httpcore/_async/http11.py", line 224, in _receive_event
    data = await self._network_stream.read(
  File "metagpt/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 36, in read
    return b""
  File "3.9.18/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "metagpt/lib/python3.9/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ReadTimeout
```
**Bug solved method**

It would be nice if the timeout and retries are configurable to avoid this issue (for example like AutoGen does this in the LLM API configuration). N.b. I've tried larger local models in the past (for which disk swapping was required due to memory constraints). Those models can sometimes take more than an hour to respond. The model for which this bug is registered can fit in my CPU RAM (64Gb).


[BUG] printing -\n- in rich.progress context manager will kill the jupyter.

try this code in the jupyter notebook:

```python
from rich.progress import Progress
with Progress() as progress:
    print("-\n-")
print("finished")
```
and it will show a popup message displaying that the kernel has died.
I have tested it on google colab and mint.

also, I have installed rich using
```
pip install rich[jupyter]
```


using sparse matrix in fit_params

When the value of a fit_params is sparse matrix, it will raise error from the following code.
sklearn/cross_validation.py

```
1224                       if hasattr(v, '__len__') and len(v) == n_samples else v)
1225                       for k, v in fit_params.items()])
```

It is because the `__len__` of sparse matrix is defined as
scipy/sparse/base.py

```
190    def __len__(self):
191        # return self.getnnz()
192        raise TypeError("sparse matrix length is ambiguous; use getnnz()"
193                         " or shape[0]")
```

Is there anyway to circumpass this issue. I do not want to convert the sparse matrix into a dense one, since it will consume a big memory.


SVC and OneVsOneClassifier decision_function inconsistent on sub-sample

Hi,

I'm seeing inconsistent numerical results with SVC's decision_function.
When estimated over an entire batch of samples ( (n_samples, n_features) matrix ) compared to analyzing sample-by-sample, the results are not the same.
This is true for both the individual numerical values per sample and the overall distribution of the results.

**The model is SVC with RBF kernel, for a 3-class classification:**
```
SVC(C=1.0, gamma=0.007, class_weight = new_class_weight, probability = True, random_state = 30,
decision_function_shape = 'ovr')
```

**The models are loaded from file:**

`ML = joblib.load("model.pkl")`

**Option A, analyze a matrix:**

`distances = ML.decision_function(X)`

**Option B, analyze individual samples:**
```
distances = numpy.zeros([X.shape[0], 3])
for i in range(X.shape[0]):
    distances[i,:]` = ML.decision_function(X[i,:].reshape(1,-1))
```

**Output for first two samples:**
**Option A:**
sample 1: [ 0.90835588, -0.17305875,  2.26470288]
sample 2: [ 1.10437313, -0.2371539 ,  2.13278077]

**Option B:**
sample 1: [ 0.82689247, -0.32689247,  2.5       ]
sample 2: [ 1.22005359, -0.5       ,  2.27994641]

I couldn't find any indication for this behavior in the documentation.

Windows-10-10.0.15063-SP0
Python 3.5.2 |Anaconda 4.2.0 (64-bit)| (default, Jul  5 2016, 11:41:13) [MSC v.1900 64 bit (AMD64)]
NumPy 1.12.1
SciPy 0.18.1
Scikit-Learn 0.18.1

Thanks!


allow spiders to return dicts instead of Items

In many cases the requirement to define and yield Items from a spider is an unnecessary complication.

An example from Scrapy tutorial:

```
import scrapy

class DmozItem(scrapy.Item):
    title = scrapy.Field()
    link = scrapy.Field()
    desc = scrapy.Field()

class DmozSpider(scrapy.Spider):
    name = "dmoz"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
    ]

    def parse(self, response):
        for sel in response.xpath('//ul/li'):
            item = DmozItem()
            item['title'] = sel.xpath('a/text()').extract()
            item['link'] = sel.xpath('a/@href').extract()
            item['desc'] = sel.xpath('text()').extract()
            yield item
```

It can be made simpler with dicts instead of Items:

```
import scrapy

class DmozSpider(scrapy.Spider):
    name = "dmoz"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
    ]

    def parse(self, response):
        for sel in response.xpath('//ul/li'):
            yield {
                'title': sel.xpath('a/text()').extract(),
                'link': sel.xpath('a/@href').extract(),
                'desc': sel.xpath('text()').extract(),
            }
```

The version with dicts gives a developer less concepts to learn, and it is easier to explain.

When field metadata is not used and data is exported to JSON/XML yielding Python dicts should be enough. Even when you export to CSV dicts could be enough - columns can be set explicitly by an user.

This should also prevent tickets like https://github.com/scrapy/scrapy/issues/968.


Duplicated input points silently create duplicated clusters in KMeans

#### Description
When there are duplicated input points to Kmeans resulting to number of unique points < number of requested clusters, there is no error thrown. Instead, clustering continues to (seemingly) produce the number of clusters requested, but some of them are exactly the same, so the cluster labels produced for the input points do not go all the way to number of requested clusters.

#### Steps/Code to Reproduce
```python
from sklearn.cluster import KMeans
import numpy as np

# some input points here are identical, so that n_total=17, n_unique=9
x2d = np.array([(1086, 348), (1087, 347), (1190, 244), (1190, 244), (1086, 348), (1185, 249), (1193, 241), (1185, 249), (1087, 347), (1188, 247), (1187, 233), (26, 111), (26, 111), (26, 110), (26, 110), (26, 110), (26, 110)])
kmeans = KMeans(n_clusters=10) # n_clusters > n_unique
c_labels = kmeans.fit_predict(x2d)
c_centers = kmeans.cluster_centers_
```
#### Expected Results
Either an error thrown, or the cluster labels produced should match the unique clusters only (i.e. no identical cluster centres)

#### Actual Results
```python
>>> c_labels  # note there's no entry for cluster 9
array([7, 2, 6, 6, 7, 5, 4, 5, 2, 1, 3, 8, 8, 0, 0, 0, 0], dtype=int32)
>>> c_centers # two of these 10 clusters have identical centers, so only 9 of them are unique
array([[   26.,   110.],
       [ 1188.,   247.],
       [ 1087.,   347.],
       [ 1187.,   233.],
       [ 1193.,   241.],
       [ 1185.,   249.],
       [ 1190.,   244.],
       [ 1086.,   348.],
       [   26.,   111.],
       [   26.,   110.]])
```

#### Versions
```python
Darwin-16.7.0-x86_64-i386-64bit
Python 3.6.1 |Continuum Analytics, Inc.| (default, May 11 2017, 13:04:09)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
NumPy 1.13.1
SciPy 0.19.1
Scikit-Learn 0.18.2
```


is_generator_with_return_value raises IndentationError with a flush left doc string

### Description

Code that is accepted by the python interpreter raises when fed through `textwrap.dedent`

### Steps to Reproduce

1. Create `is_generator_bug.py` with the content below (which I simplified from [the `is_generator_with_return_value` method body](https://github.com/scrapy/scrapy/blob/2.0.1/scrapy/utils/misc.py#L186-L187)
2. Run `python is_generator_bug.py`
3. Observe the kaboom

```python
import ast
import inspect
from textwrap import dedent
class Bob:
    def doit(self):
        """
this line is flush left
        """
        if True:
            yield 1234

if __name__ == '__main__':
    b = Bob()
    c = b.doit
    if inspect.isgeneratorfunction(c):
        tree = ast.parse(dedent(inspect.getsource(c)))
```

**Expected behavior:** [What you expect to happen]

No Error

**Actual behavior:** [What actually happens]

```console
$ python3.7 is_generator_bug.py
Traceback (most recent call last):
  File "is_generator_bug.py", line 16, in <module>
    tree = ast.parse(dedent(inspect.getsource(c)))
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 1
    def doit(self):
    ^
IndentationError: unexpected indent
```

**Reproduces how often:** [What percentage of the time does it reproduce?]

100%

### Versions

```
Scrapy       : 2.0.1
lxml         : 4.5.0.0
libxml2      : 2.9.10
cssselect    : 1.1.0
parsel       : 1.5.2
w3lib        : 1.21.0
Twisted      : 20.3.0
Python       : 3.7.7 (default, Mar 11 2020, 23:30:22) - [Clang 10.0.0 (clang-1000.11.45.5)]
pyOpenSSL    : 19.1.0 (OpenSSL 1.1.1d  10 Sep 2019)
cryptography : 2.8
Platform     : Darwin-17.7.0-x86_64-i386-64bit
```

### Additional context


Something goes wrong with KernelPCA with 32 bits input data

When given 32 bits input, KernelPCA succeed to transform the data into a 17-dimensional feature space while the original space was 3 features. I did not debug yet but this seems really unlikely.

```python
# %%
from sklearn.datasets import make_blobs
from sklearn.preprocessing import StandardScaler

X, y = make_blobs(
    n_samples=30,
    centers=[[0, 0, 0], [1, 1, 1]],
    random_state=0,
    cluster_std=0.1
)
X = StandardScaler().fit_transform(X)
X -= X.min()

# %%
import numpy as np
from sklearn.decomposition import KernelPCA

kpca = KernelPCA()
print(kpca.fit_transform(X).shape)
print(kpca.fit_transform(X.astype(np.float32)).shape)
```


S3 object metadata not saved when uploaded with presigned url

Use case:
I'm enabling users to directly upload to s3 using presigned url. S3 is configured to add event to SQS on Put. Queue consumer, reads the queue and makes HEAD requests with object keys to get the metadata and save information to database (generic image upload, so I know where to add file).

Test script in node js - some ugly code here (to install deps run `npm install aws-sdk request`):
```js
const AWS = require("aws-sdk");
const request = require("request");

let s3 = new AWS.S3({
    endpoint: "http://localhost:4572",
    s3ForcePathStyle: true,
    accessKeyId: "",
    secretAccessKey: "",
    region: "us-west-1"
});

var bucket = "bucketest";
var key = "test.txt";

s3.createBucket({Bucket: bucket}, function (err, data) {
    if (err) {
        console.error(err.message);
        // ignore, probably there is bucket already
    }

    var params = {
        Bucket: bucket,
        Key: key,
        Metadata: {
            venue: "123"
        }
    };

    s3.getSignedUrl('putObject', params, function (err, url) {
        if (err) {
            console.error('Presigning post data encountered an error', err);
        } else {
            console.log('==== URL: ', url);

            var body = new Buffer('Test data.');
            request.put({ url, body, method: "PUT" }, function(err, resp, body) {
                if (err) {
                    console.log('======= error:', error);
                    return;
                }

                console.log(body);

                s3.headObject({Bucket: bucket, Key: key}, function (err, data) {
                    if (err) console.log("====== error1:", err, err.stack);
                    else console.log("==== HEAD RESPONSE", data);
                });
            })
        }
    });
});
```

Output:
```
==== URL:  http://localhost:4572/heaps-test/test.txt?AWSAccessKeyId=somekey&Expires=1515503310&Signature=TgK3B33p2kwCWs5F5KtaZ3fxgXA%3D&x-amz-meta-venue=123
<PutObjectResponse xmlns="http://s3.amazonaws.com/doc/2006-03-01"><PutObjectResponse><ETag>"56dd8a439abf97fda051f88f09f00d65"</ETag><LastModified>2018-01-09T12:53:30.637Z</LastModified></PutObjectResponse></PutObjectResponse>
==== HEAD RESPONSE { LastModified: 2018-01-09T12:53:30.000Z,
  ContentLength: 10,
  ETag: '"56dd8a439abf97fda051f88f09f00d65"',
  ContentType: 'text/html; charset=utf-8',
  Metadata: {} }

```

Expected Output (tested with live AWS):
```
==== URL:  https://heaps-test.s3.eu-west-1.amazonaws.com/test.txt?AWSAccessKeyId=somekey&Expires=1515503234&Signature=enc17C6glTsVtOiGobugz5NELIc%3D&x-amz-meta-venue=123

==== HEAD RESPONSE { AcceptRanges: 'bytes',
  LastModified: 2018-01-09T12:52:15.000Z,
  ContentLength: 10,
  ETag: '"56dd8a439abf97fda051f88f09f00d65"',
  ContentType: 'binary/octet-stream',
  Metadata: { venue: '123' } }
```

As you can see Metadata is empty when using localstack


This model's maximum context length is 8191 tokens, however you requested 89686 tokens (89686 in your prompt)

### Duplicates

- [X] I have searched the existing issues

### Steps to reproduce 🕹

The program is trying to process an absurd amount of information at once. It happens over and over again.

Adding chunk 17 / 20 to memory
SYSTEM:  Command browse_website returned: Error: This model's maximum context length is 8191 tokens, however you requested 89686 tokens (89686 in your prompt;
 0 for the completion). Please reduce your prompt; or completion length.

### Current behavior 😯

_No response_

### Expected behavior 🤔

_No response_

### Your prompt 📝

```yaml
# Paste your prompt here
```


stat documentation: mime_type vs mimetype, mime output vs descriptive output

<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and devel branch are affected too.
-->

##### ISSUE TYPE
 - Documentation Report

##### COMPONENT NAME
stat

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
ansible 2.4.3.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/oliver/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]
```

##### CONFIGURATION
ansible.cfg:
```
[defaults]
inventory      = ./hosts
roles_path     = ./roles
remote_user    = oliver
nocows         = 1
vault_password_file = ./get_vault_password_from_keyring.py

gathering      = smart
fact_caching   = jsonfile
fact_caching_timeout = 21600
fact_caching_connection = ./cache

[privilege_escalation]
become = True
become_method = sudo

[ssh_connection]
pipelining = True
```

##### OS / ENVIRONMENT
Running Ansible on Ubuntu 16.04.4 x86_64; target is the same machine.

Tried with and without python-magic module installed, but didn't observe any different behaviour.

##### SUMMARY
<!--- Explain the problem briefly -->
Documentation for stat module (http://docs.ansible.com/ansible/latest/stat_module.html) mentions that a "mime_type" entry will be set if get_mime is set to true, with example content being "PDF document, version 1.2". I couldn't get this result; rather:
- a "mimetype" entry is set (ie. no underscore)
- the mimetype entry contains the actual mime type (eg. "application/pdf") rather than a description (eg. "PDF document, version 1.2")

##### STEPS TO REPRODUCE
```yaml
- name: Get mime type of test file
  stat: path="/home/oliver/mozilla.pdf"
  register: my_stat_check

- debug:
    msg: "{{ my_stat_check.stat.mimetype }}"

- debug:
    msg: "{{ my_stat_check.stat.mime_type }}"
```

##### EXPECTED RESULTS
I expected the first debug message to fail, and expected the second one to succeed and print "PDF document, version 1.2".

Alternatively, the documentation should state that "mimetype" will be set, and will contain the technical mime type rather than a description.
Though admittedly I'd prefer to also get the descriptive type output, since eg. for swap files the mime type is always "application/octet-stream" (so a swap file is indistinguishable from any other binary file); while the descriptive type is something like "Linux/i386 swap file (new style)" which is more useful.

##### ACTUAL RESULTS
At the moment, the first debug message will work and will print "application/pdf". The second debug message will fail with "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'mime_type'".

```
TASK [tools : Get mime type of test file] *****************************************************************************************************************************************************************************************************
ok: [myhost] => {"changed": false, "stat": {"atime": 1519571973.3959274, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 352, "charset": "binary", "checksum": "2d9eb9f17601726c56bd0c4fbc770430d0ac2277", "ctime": 1519484291.234419, "dev": 2098, "device_type": 0, "executable": false, "exists": true, "gid": 1000, "gr_name": "oliver", "inode": 12592144, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "md5": "760bf09b20afc699ad5cb4cabf3a151a", "mimetype": "application/pdf", "mode": "0664", "mtime": 1519484291.234419, "nlink": 1, "path": "/home/oliver/mozilla.pdf", "pw_name": "oliver", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 180168, "uid": 1000, "version": "18446744071694658390", "wgrp": true, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}}

TASK [tools : debug] **************************************************************************************************************************************************************************************************************************
ok: [myhost] => {
    "msg": "application/pdf"
}

TASK [tools : debug] **************************************************************************************************************************************************************************************************************************
fatal: [myhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'mime_type'\n\nThe error appears to have been in '/home/oliver/devel/myhost/ansible/roles/tools/tasks/main.yml': line 40, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- debug:\n  ^ here\n\nexception type: <class 'ansible.errors.AnsibleUndefinedVariable'>\nexception: 'dict object' has no attribute 'mime_type'"}
	to retry, use: --limit @/home/oliver/devel/myhost/ansible/playbooks/desktop.retry
```


kindly adding some documentations on t5-v1_1-base""

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

Documentation: @sgugger
Hi
Could you kindly add some documentations on "t5-v1_1-base"? I tested one code with t5-base and t5-v1 version, for t5-v1 I got memory issue, this seems to me the model size is different and larger, also fast tokenizer for this model does not work, could you kindly add a documentation on these differences?

thanks a lot.