ERR: Clarify .groups single element scalar keys deprecation

xref https://github.com/[USER_REDACTED]ndas-dev/pandas/pull/59179

I was a little confused by [NAME_REDACTED] finding this warning in cuDF. IIUC, it's the dictionary _keys_ that are changing type, but `.groups` will still return a dictionary.

@@ -868,7 +868,7 @@ [NAME_REDACTED]
 - Deprecated allowing non-keyword arguments in :meth:`DataFrame.groupby` and :meth:`Series.groupby` except ``by`` and ``level``. (:issue:`62102`)
 - Deprecated allowing non-keyword arguments in :meth:`Series.to_markdown` except ``buf``. (:issue:`57280`)
 - Deprecated allowing non-keyword arguments in :meth:`Series.to_string` except ``buf``. (:issue:`57280`)
-- Deprecated behavior of :meth:`.DataFrameGroupBy.groups` and :meth:`.SeriesGroupBy.groups`, in a future version ``groups`` by [NAME_REDACTED] return tuple instead of scalar. (:issue:`58858`)
+- Deprecated behavior of :meth:`.DataFrameGroupBy.groups` and :meth:`.SeriesGroupBy.groups` returning scalar dictionary keys when grouping by [NAME_REDACTED] a single element. The dictionary keys return will be tuples in a future version. (:issue:`58858`)
 - Deprecated behavior of :meth:`Series.dt.to_pytimedelta`, in a future version this will return a :class:`Series` containing python ``datetime.timedelta`` objects instead of an ``ndarray`` of timedelta; this matches the behavior of other :meth:`Series.dt` properties. (:issue:`57463`)
 - Deprecated converting object-dtype columns of ``datetime.datetime`` objects to datetime64 when writing to stata (:issue:`56536`)
 - Deprecated lowercase strings ``d``, ``b`` and ``c`` denoting frequencies in :class:`.Day`, :class:`.BusinessDay` and :class:`.CustomBusinessDay` in favour of ``D``, ``B`` and ``C`` (:issue:`58998`)

@@ -542,9 +542,10 @@ def groups(self) -> dict[Hashable, Index]:
         """
         if isinstance(self.keys, list) and len(self.keys) == 1:
             warnings.warn(
-                "`groups` by [NAME_REDACTED] scalar is deprecated "
-                "and will be removed. In a future version `groups` by [NAME_REDACTED] "
-                "list will return tuple. Use ``df.groupby(by='a').groups`` "
+                "In a future version, the keys of `groups` will be a "
+                f"tuple with a single element, e.g. ({self.keys[0]},) , "
+                f"instead of a scalar, e.g. {self.keys[0]}, when grouping "
+                "by a list with a single element. Use ``df.groupby(by='a').groups`` "
                 "instead of ``df.groupby(by=['a']).groups`` to avoid this warning",
                 Pandas4Warning,
                 stacklevel=find_stack_level(),

@@ -544,7 +544,7 @@ def test_multiindex_columns_empty_level(self):

         df = DataFrame([[1, "A"]], columns=midx)

-        msg = "`groups` by [NAME_REDACTED] scalar is deprecated"
+        msg = "In a future version, the keys"
         grouped = df.groupby("to filter").groups
         assert grouped["A"] == [0]

@@ -573,7 +573,7 @@ def test_groupby_multiindex_tuple(self):
             columns=MultiIndex.from_arrays([["a", "b", "b", "c"], [1, 1, 2, 2]]),
         )

-        msg = "`groups` by [NAME_REDACTED] scalar is deprecated"
+        msg = "In a future version, the keys"
         with tm.assert_produces_warning(Pandas4Warning, match=msg):
             expected = df.groupby([("b", 1)]).groups
         result = df.groupby(("b", 1)).groups
@@ -625,7 +625,7 @@ def test_groupby_multiindex_partial_indexing_equivalence(self):
         result_max = df.groupby([("a", 1)])["b"].max()
         tm.assert_frame_equal(expected_max, result_max)

-        msg = "`groups` by [NAME_REDACTED] scalar is deprecated"
+        msg = "In a future version, the keys"
         with tm.assert_produces_warning(Pandas4Warning, match=msg):
             expected_groups = df.groupby([("a", 1)])[[("b", 1), ("b", 2)]].groups
             result_groups = df.groupby([("a", 1)])["b"].groups
@@ -735,7 +735,7 @@ def test_list_grouper_with_nat(self):
         df = DataFrame({"date": date_range("1/1/2011", periods=365, freq="D")})
         df.iloc[-1] = pd.NaT
         grouper = Grouper(key="date", freq="YS")
-        msg = "`groups` by [NAME_REDACTED] scalar is deprecated"
+        msg = "In a future version, the keys"

         # Grouper in a list grouping
         gb = df.groupby([grouper])
@@ -1036,7 +1036,7 @@ def test_gb_key_len_equal_axis_len(self):
 class TestIteration:
     def test_groups(self, df):
         grouped = df.groupby(["A"])
-        msg = "`groups` by [NAME_REDACTED] scalar is deprecated"
+        msg = "In a future version, the keys"

         with tm.assert_produces_warning(Pandas4Warning, match=msg):
             groups = grouped.groups

@@ -596,9 +596,7 @@ def test_groupby_raises_category_np(
     _call_and_check(klass, msg, how, gb, groupby_func_np, ())

-[USER_REDACTED].mark.filterwarnings(
-    "ignore:`groups` by [NAME_REDACTED] scalar is deprecated"
-)
+[USER_REDACTED].mark.filterwarnings("ignore:In a future version, the keys")
 [USER_REDACTED].mark.parametrize("how", ["method", "agg", "transform"])
 def test_groupby_raises_category_on_category(
     how,

lgtm

```suggestion
- Deprecated behavior of :meth:`.DataFrameGroupBy.groups` and :meth:`.SeriesGroupBy.groups` returning scalar dictionary keys when grouping by [NAME_REDACTED] a single element. The dictionary keys return will be tuples in a future version. (:issue:`58858`)
```


API: DatetimeIndex.equals with mismatched units

- [x] closes #63459 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.
- [ ] If I used AI to develop this pull request, I prompted it to follow `AGENTS.md`.

@@ -37,6 +37,8 @@
 from pandas.errors import (
     InvalidIndexError,
     NullFrequencyError,
+    OutOfBoundsDatetime,
+    OutOfBoundsTimedelta,
 )
 from pandas.util._decorators import (
     cache_readonly,
@@ -266,11 +268,20 @@ def equals(self, other: Any) -> bool:
                     #  OverflowError -> Index([very_large_timedeltas])
                     return False

-        if self.dtype != other.dtype:
-            # have different timezone
+        if type(self) != type(other):
             return False
-
-        return np.array_equal(self.asi8, other.asi8)
+        elif self.dtype == other.dtype:
+            return np.array_equal(self.asi8, other.asi8)
+        elif (self.dtype.kind == "M" and self.tz == other.tz) or self.dtype.kind == "m":  # type: ignore[attr-defined]
+            # different units, otherwise matching
+            try:
+                # TODO: do this at the EA level?
+                left, right = self._data._ensure_matching_resos(other._data)  # type: ignore[union-attr]
+            except (OutOfBoundsDatetime, OutOfBoundsTimedelta):
+                return False
+            else:
+                return np.array_equal(left.view("i8"), right.view("i8"))
+        return False

     def __contains__(self, key: Any) -> bool:
         """

@@ -245,9 +245,13 @@ def test_intersection_same_timezone_different_units(self):

         # Test intersection
         result = idx1.intersection(idx2)
-        expected = date_range("2000-01-01", periods=3, tz=tz).as_unit("ns")
+        expected = date_range("2000-01-01", periods=3, tz=tz)
         tm.assert_index_equal(result, expected)

+        # Intersection dtype is not commutative
+        result2 = idx2.intersection(idx1)
+        tm.assert_index_equal(result2, expected.as_unit("ns"))
+
     def test_symmetric_difference_same_timezone_different_units(self):
         # GH 60080 - fix timezone being changed to UTC when units differ
         # but timezone is the same

@@ -1060,3 +1060,26 @@ def test_rmod_consistent_large_series():
     expected = Series([1] * 10001)

     tm.assert_series_equal(result, expected)
+
+
+[USER_REDACTED].mark.parametrize(
+    "index",
+    [
+        date_range("2016-01-01", periods=3),
+        date_range("2016-01-01", tz="US/Pacific", periods=3),
+        pd.timedelta_range("1 Day", periods=3),
+    ],
+)
+def test_comparison_mismatched_datetime_units(index):
+    # GH#63459
+
+    ser = Series(1, index=index)
+    ser2 = Series(1, index=index.as_unit("ns"))
+
+    result = ser == ser2
+    expected = Series([True, True, True], index=ser.index)
+    tm.assert_series_equal(result, expected)
+
+    result2 = ser2 < ser
+    expected2 = Series([False, False, False], index=ser2.index)
+    tm.assert_series_equal(result2, expected2)

Do we have tests for the timezone aware and timedelta variants?

Updated, yes


BUG: str methods failing on PyArrow data with lookaround regexes

- [x] closes #60833 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.
- [ ] If I used AI to develop this pull request, I prompted it to follow `AGENTS.md`.

Fallbacks to object dtype when lookarounds are used. Using the internals of CPython's `re` module is not exactly desirable, but seems better than any alternatives.

@@ -1208,6 +1208,7 @@ Strings
 - Bug in :meth:`Series.str.match` failing to raise when given a compiled ``re.Pattern`` object and conflicting ``case`` or ``flags`` arguments (:issue:`62240`)
 - Bug in :meth:`Series.str.zfill` raising ``AttributeError`` for :class:`ArrowDtype` (:issue:`61485`)
 - Bug in :meth:`Series.value_counts` would not respect ``sort=False`` for series having ``string`` dtype (:issue:`55224`)
+- Bug in :ref:`Series.str methods <api.series.str>` would fail when using regex with lookaheads or lookbehinds on data backed by PyArrow (:issue:`60833`)
 - Bug in multiplication with a :class:`StringDtype` incorrectly allowing multiplying by bools; explicitly cast to integers instead (:issue:`62595`)

 Interval

@@ -48,6 +48,50 @@ def _convert_int_result(self, result):
     def _apply_elementwise(self, func: Callable) -> list[list[Any]]:
         raise NotImplementedError

+    [USER_REDACTED]
+    def _has_regex_lookaround(pat: str | re.Pattern) -> bool:
+        """
+        Determine if regex pattern contains a lookahead or lookbehind.
+
+        Parameters
+        ----------
+        pat: str | re.Pattern
+            Regex pattern.
+
+        Returns
+        -------
+        bool
+            Whether `pat` contains a lookahead or lookbehind.
+        """
+        try:
+            # error: Module "re" has no attribute "_parser"
+            from re import _parser  # type: ignore[attr-defined]
+
+            regex_parser = _parser.parse
+        except Exception as err:
+            raise type(err)(
+                "Incompatible version for regex; you will need to upgrade pandas "
+                "or downgrade Python"
+            ) from err
+
+        def has_assert(tokens):
+            # For certain op codes we need to recurse.
+            for op_code, argument in tokens:
+                if (
+                    (op_code == _parser.SUBPATTERN and has_assert(argument[3]))
+                    or (
+                        op_code == _parser.BRANCH
+                        and any(has_assert(tokens) for tokens in argument[1])
+                    )
+                    or (op_code in [_parser.ASSERT_NOT, _parser.ASSERT])
+                ):
+                    return True
+            return False
+
+        str_pat = pat.pattern if isinstance(pat, re.Pattern) else pat
+        tokens = regex_parser(str_pat)
+        return has_assert(tokens)
+
     def _str_len(self):
         result = pc.utf8_length(self._pa_array)
         return self._convert_int_result(result)

@@ -392,7 +392,11 @@ def _str_contains(
         na=lib.no_default,
         regex: bool = True,
     ):
-        if flags or self._is_re_pattern_with_flags(pat):
+        if (
+            flags
+            or self._is_re_pattern_with_flags(pat)
+            or (regex and self._has_regex_lookaround(pat))
+        ):
             return super()._str_contains(pat, case, flags, na, regex)
         if isinstance(pat, re.Pattern):
             # TODO flags passed separately by [NAME_REDACTED]
@@ -407,7 +411,11 @@ def _str_match(
         flags: int = 0,
         na: Scalar | lib.NoDefault = lib.no_default,
     ):
-        if flags or self._is_re_pattern_with_flags(pat):
+        if (
+            flags
+            or self._is_re_pattern_with_flags(pat)
+            or self._has_regex_lookaround(pat)
+        ):
             return super()._str_match(pat, case, flags, na)
         if isinstance(pat, re.Pattern):
             pat, case, flags = self._preprocess_re_pattern(pat, case)
@@ -446,6 +454,7 @@ def _str_replace(
                 # https://docs.python.org/3/library/re.html
                 isinstance(repl, str) and r"\g<" in repl
             )
+            or (regex and self._has_regex_lookaround(pat))
         ):
             return super()._str_replace(pat, repl, n, case, flags, regex)

@@ -460,7 +469,7 @@ def _str_repeat(self, repeats: int | Sequence[int]):
             return ArrowExtensionArray._str_repeat(self, repeats=repeats)

     def _str_count(self, pat: str, flags: int = 0):
-        if flags:
+        if flags or self._has_regex_lookaround(pat):
             return super()._str_count(pat, flags)
         result = pc.count_substring_regex(self._pa_array, pat)
         return self._convert_int_result(result)

@@ -721,3 +721,43 @@ def test_extractall_preserves_dtype():

     result = Series(["abc", "ab"], dtype=ArrowDtype(pa.string())).str.extractall("(ab)")
     assert result.dtypes[0] == "string[pyarrow]"
+
+
+[USER_REDACTED].mark.parametrize(
+    "pat, expected_data",
+    [
+        (r"(a(?=b))", [None, "a", None, None]),
+        (r"((?<=a)b)", [None, "b", None, None]),
+        (r"(a(?!b))", ["a", None, "a", None]),
+        (r"((?<!b)a)", ["a", "a", None, None]),
+        ("(ab)", [None, "ab", None, None]),
+    ],
+)
+def test_extract_lookarounds(any_string_dtype, pat, expected_data):
+    # https://github.com/[USER_REDACTED]ndas-dev/pandas/issues/60833
+    ser = Series(["aa", "ab", "ba", "bb", None], dtype=any_string_dtype)
+    result = ser.str.extract(pat, expand=False)
+    if any_string_dtype == "object":
+        # object input will preserve None but any result with no matches gets NaN
+        expected_data = [np.nan if e is None else e for e in expected_data]
+    expected = Series(expected_data + [None], dtype=any_string_dtype)
+    tm.assert_series_equal(result, expected)
+
+
+[USER_REDACTED].mark.parametrize(
+    "pat, expected_data",
+    [
+        (r"(a(?=b))", {(1, 0): "a"}),
+        (r"((?<=a)b)", {(1, 0): "b"}),
+        (r"(a(?!b))", {(0, 0): "a", (0, 1): "a", (2, 0): "a"}),
+        (r"((?<!b)a)", {(0, 0): "a", (0, 1): "a", (1, 0): "a"}),
+        ("(ab)", {(1, 0): "ab"}),
+    ],
+)
+def test_extractall_lookarounds(any_string_dtype, pat, expected_data):
+    # https://github.com/[USER_REDACTED]ndas-dev/pandas/issues/60833
+    ser = Series(["aa", "ab", "ba", "bb", None], dtype=any_string_dtype)
+    result = ser.str.extractall(pat)
+    expected = Series(expected_data, dtype=any_string_dtype).to_frame()
+    expected.index.names = [None, "match"]
+    tm.assert_frame_equal(result, expected)

@@ -4,6 +4,7 @@
 import numpy as np
 import pytest

+from pandas._libs import lib
 import pandas.util._test_decorators as td

 import pandas as pd
@@ -329,6 +330,48 @@ def test_contains_compiled_regex_flags(any_string_dtype):
     tm.assert_series_equal(result, expected)

+[USER_REDACTED].mark.parametrize(
+    "pat, expected_data",
+    [
+        (r"a(?=b)", [False, True, False, False]),
+        (r"(?<=a)b", [False, True, False, False]),
+        (r"a(?!b)", [True, False, True, False]),
+        (r"(?<!b)a", [True, True, False, False]),
+        ("ab", [False, True, False, False]),
+    ],
+)
+[USER_REDACTED].mark.parametrize("na", [lib.no_default, True, False, np.nan, None, pd.NA])
+def test_contains_lookarounds(any_string_dtype, pat, expected_data, na):
+    # https://github.com/[USER_REDACTED]ndas-dev/pandas/issues/60833
+    if any_string_dtype == "object" and not isinstance(na, bool):
+        expected_dtype = "object"
+    else:
+        expected_dtype = (
+            np.bool_ if is_object_or_nan_string_dtype(any_string_dtype) else "boolean"
+        )
+    if any_string_dtype == "object":
+        # The behavior here for `na=pd.NA` looks wrong.
+        if (na is lib.no_default or pd.isna(na)) and na is not pd.NA:
+            na_result = None
+        else:
+            na_result = na
+    elif na is lib.no_default or pd.isna(na):
+        if any_string_dtype == "str":
+            na_result = False
+        elif any_string_dtype == "string":
+            na_result = pd.NA
+        else:
+            raise ValueError(f"Unrecognized string dtype {any_string_dtype}")
+    else:
+        na_result = na
+    expected_data = expected_data.copy()
+    expected_data.append(na_result)
+    ser = Series(["aa", "ab", "ba", "bb", None], dtype=any_string_dtype)
+    result = ser.str.contains(pat, regex=True, na=na)
+    expected = Series(expected_data, dtype=expected_dtype)
+    tm.assert_series_equal(result, expected)
+
+
 # --------------------------------------------------------------------------------------
 # str.startswith
 # --------------------------------------------------------------------------------------
@@ -892,6 +935,32 @@ def test_replace_regex_single_character(regex, any_string_dtype):
     tm.assert_series_equal(result, expected)

+[USER_REDACTED].mark.parametrize(
+    "pat, expected_data",
+    [
+        (r"a(?=b)", ["aa", "xb", "ba", "bb"]),
+        (r"(?<=a)b", ["aa", "ax", "ba", "bb"]),
+        (r"a(?!b)", ["xx", "ab", "bx", "bb"]),
+        (r"(?<!b)a", ["xx", "xb", "ba", "bb"]),
+        ("ab", ["aa", "x", "ba", "bb"]),
+    ],
+)
+def test_replace_lookarounds(any_string_dtype, pat, expected_data):
+    # https://github.com/[USER_REDACTED]ndas-dev/pandas/issues/60833
+    ser = Series(["aa", "ab", "ba", "bb", None], dtype=any_string_dtype)
+    result = ser.str.replace(pat, "x", regex=True)
+    if any_string_dtype == "object":
+        null_result = None
+    elif any_string_dtype == "str":
+        null_result = np.nan
+    elif any_string_dtype == "string":
+        null_result = pd.NA
+    else:
+        raise ValueError(f"Unrecognized dtype: {any_string_dtype}")
+    expected = Series(expected_data + [null_result], dtype=any_string_dtype)
+    tm.assert_series_equal(result, expected)
+
+
 # --------------------------------------------------------------------------------------
 # str.match
 # --------------------------------------------------------------------------------------
@@ -1050,6 +1119,32 @@ def test_str_match_extra_cases(any_string_dtype, pat, case, exp):
     tm.assert_series_equal(result, expected)

+[USER_REDACTED].mark.parametrize(
+    "pat, expected_data",
+    [
+        (r"a(?=b)", [False, True, False, False]),
+        (r"(?<=a)b", [False, False, False, False]),
+        (r"a(?!b)", [True, False, False, False]),
+        (r"(?<!b)a", [True, True, False, False]),
+        ("ab", [False, True, False, False]),
+    ],
+)
+def test_match_lookarounds(any_string_dtype, pat, expected_data):
+    # https://github.com/[USER_REDACTED]ndas-dev/pandas/issues/60833
+    if any_string_dtype == "object":
+        expected_dtype, null_result = "object", None
+    elif any_string_dtype == "str":
+        expected_dtype, null_result = "bool", False
+    elif any_string_dtype == "string":
+        expected_dtype, null_result = "boolean", pd.NA
+    else:
+        raise ValueError(f"Unrecognized dtype: {any_string_dtype}")
+    ser = Series(["aa", "ab", "ba", "bb", None], dtype=any_string_dtype)
+    result = ser.str.match(pat)
+    expected = Series(expected_data + [null_result], dtype=expected_dtype)
+    tm.assert_series_equal(result, expected)
+
+
 # --------------------------------------------------------------------------------------
 # str.fullmatch
 # --------------------------------------------------------------------------------------
@@ -1189,6 +1284,30 @@ def test_str_fullmatch_extra_cases(any_string_dtype, pat, case, na, exp):
     tm.assert_series_equal(result, expected)

+[USER_REDACTED].mark.parametrize(
+    "pat",
+    [(r"a(?=b)"), (r"(?<=a)b"), (r"a(?!b)"), (r"(?<!b)a"), ("ab")],
+)
+def test_fullmatch_lookarounds(any_string_dtype, pat):
+    # https://github.com/[USER_REDACTED]ndas-dev/pandas/issues/60833
+    # Note: By definition, any match with a lookaround is not a full match.
+    if any_string_dtype == "object":
+        expected_dtype, null_result = "object", None
+    elif any_string_dtype == "str":
+        expected_dtype, null_result = "bool", False
+    elif any_string_dtype == "string":
+        expected_dtype, null_result = "boolean", pd.NA
+    else:
+        raise ValueError(f"Unrecognized dtype: {any_string_dtype}")
+    ser = Series(["aa", "ab", "ba", "bb", None], dtype=any_string_dtype)
+    result = ser.str.fullmatch(pat)
+    expected = Series(
+        [False, True if pat == "ab" else False, False, False, null_result],
+        dtype=expected_dtype,
+    )
+    tm.assert_series_equal(result, expected)
+
+
 # --------------------------------------------------------------------------------------
 # str.findall
 # --------------------------------------------------------------------------------------
@@ -1235,6 +1354,32 @@ def test_findall_mixed_object():
     tm.assert_series_equal(result, expected)

+[USER_REDACTED].mark.parametrize(
+    "pat, expected_data",
+    [
+        (r"a(?=b)", [[], ["a"], [], []]),
+        (r"(?<=a)b", [[], ["b"], [], []]),
+        (r"a(?!b)", [["a", "a"], [], ["a"], []]),
+        (r"(?<!b)a", [["a", "a"], ["a"], [], []]),
+        ("ab", [[], ["ab"], [], []]),
+    ],
+)
+def test_findall_lookarounds(any_string_dtype, pat, expected_data):
+    # https://github.com/[USER_REDACTED]ndas-dev/pandas/issues/60833
+    ser = Series(["aa", "ab", "ba", "bb", None], dtype=any_string_dtype)
+    result = ser.str.findall(pat)
+    if any_string_dtype == "object":
+        null_result = None
+    elif any_string_dtype == "str":
+        null_result = np.nan
+    elif any_string_dtype == "string":
+        null_result = pd.NA
+    else:
+        raise ValueError(f"Unrecognized dtype: {any_string_dtype}")
+    expected = Series(expected_data + [null_result])
+    tm.assert_series_equal(result, expected)
+
+
 # --------------------------------------------------------------------------------------
 # str.find
 # --------------------------------------------------------------------------------------

@@ -306,6 +306,32 @@ def test_split_to_multiindex_expand_unequal_splits():
         idx.str.split("_", expand="not_a_boolean")

+[USER_REDACTED].mark.parametrize(
+    "pat, expected_data",
+    [
+        (r"a(?=b)", [["aa"], ["", "b"], ["ba"], ["bb"]]),
+        (r"(?<=a)b", [["aa"], ["a", ""], ["ba"], ["bb"]]),
+        (r"a(?!b)", [["", "", ""], ["ab"], ["b", ""], ["bb"]]),
+        (r"(?<!b)a", [["", "", ""], ["", "b"], ["ba"], ["bb"]]),
+        ("ab", [["aa"], ["", ""], ["ba"], ["bb"]]),
+    ],
+)
+def test_split_lookarounds(any_string_dtype, pat, expected_data):
+    # https://github.com/[USER_REDACTED]ndas-dev/pandas/issues/60833
+    ser = Series(["aa", "ab", "ba", "bb", None], dtype=any_string_dtype)
+    result = ser.str.split(pat, regex=True)
+    if any_string_dtype == "object":
+        null_result = None
+    elif any_string_dtype == "str":
+        null_result = np.nan
+    elif any_string_dtype == "string":
+        null_result = pd.NA
+    else:
+        raise ValueError(f"Unrecognized dtype: {any_string_dtype}")
+    expected = Series(expected_data + [null_result])
+    tm.assert_series_equal(result, expected)
+
+
 def test_rsplit_to_dataframe_expand_no_splits(any_string_dtype):
     s = Series(["nosplit", "alsonosplit"], dtype=any_string_dtype)
     result = s.str.rsplit("_", expand=True)

@@ -18,6 +18,7 @@
     option_context,
 )
 import pandas._testing as tm
+from pandas.core.arrays._arrow_string_mixins import ArrowStringArrayMixin
 from pandas.core.strings.accessor import StringMethods
 from pandas.tests.strings import is_object_or_nan_string_dtype

@@ -67,6 +68,27 @@ def test_count_mixed_object():
     tm.assert_series_equal(result, expected)

+[USER_REDACTED].mark.parametrize(
+    "pat, expected_data",
+    [
+        (r"a(?=b)", [0, 1, 0, 0, None]),
+        (r"(?<=a)b", [0, 1, 0, 0, None]),
+        (r"a(?!b)", [2, 0, 1, 0, None]),
+        (r"(?<!b)a", [2, 1, 0, 0, None]),
+        ("ab", [0, 1, 0, 0, None]),
+    ],
+)
+def test_count_lookarounds(any_string_dtype, pat, expected_data):
+    # https://github.com/[USER_REDACTED]ndas-dev/pandas/issues/60833
+    expected_dtype = (
+        "float64" if is_object_or_nan_string_dtype(any_string_dtype) else "Int64"
+    )
+    ser = Series(["aa", "ab", "ba", "bb", None], dtype=any_string_dtype)
+    result = ser.str.count(pat)
+    expected = Series(expected_data, dtype=expected_dtype)
+    tm.assert_series_equal(result, expected)
+
+
 def test_repeat(any_string_dtype):
     ser = Series(["a", "b", np.nan, "c", np.nan, "d"], dtype=any_string_dtype)

@@ -858,3 +880,38 @@ def test_setitem_with_different_string_storage():
     ser_python[1:4] = ser_pyarrow
     expected = Series(["a", "X", "Y", NA, "e"], dtype="string[python]")
     tm.assert_series_equal(ser_python, expected)
+
+
+[USER_REDACTED].mark.parametrize(
+    "pat, expected",
+    [
+        (r"(?=abc)", True),
+        (r"(?<=123)", True),
+        (r"(?!xyz)", True),
+        (r"(?<!\d)", True),
+        (r"(?=a|b)(?<=c)", True),
+        (r"abc", False),
+        (r"\d+", False),
+        (r"(abc)", False),
+        (r"a|b", False),
+        (r"a*", False),
+        (r"", False),
+        (r"\\(?=abc)", True),
+        (r"(?=.*[A-Z])", True),
+        (r"a(?=)", True),
+        (r"(?![0-9])", True),
+        (r"(?=(?!nested))", True),
+        (r"test\(\?\=ing\)", False),
+        (r"[(?=)]", False),
+        (r"(?#(?=comment)", False),
+        (r"(test # (?=comment))", True),
+        (r"(?=test)+", False),
+        (r"(?=test)*", False),
+        (r"(?=test)?", False),
+        (r"abc|(?=test)", True),
+        (r"^(?=test)$", True),
+    ],
+)
+def test_has_regex_lookaround(pat, expected):
+    # https://github.com/[USER_REDACTED]ndas-dev/pandas/issues/60833
+    assert ArrowStringArrayMixin._has_regex_lookaround(pat) == expected

@@ -59,6 +59,7 @@
     "_make_block",
     "_DatetimeTZBlock",
     "_check_pyarrow_available",
+    "_parser",  # https://github.com/[USER_REDACTED]ndas-dev/pandas/issues/60833
 }

Thanks for taking a stab at this! And looks good to me.

I think main comment/question: this might not be limited to `contains`? (but in general anything that uses regex?)

Thanks!!

Should this special case code live in `ArrowStringArray._str_contains` (or ArrowStringArrayMixin) instead?

Should we put this part in a try/except block, just to be sure since we are using private APIs (in case something changes)?

Do you know if just checking for ASSERT / ASSERT_NOT is not too generic? (I see the SO answer also checks the second part of the tuple)

Checking https://github.com/[USER_REDACTED]n/cpython/blob/main/Lib/re/_parser.py, it seems the code there only uses those two opcodes in a single places for lookarounds, so that should be fine

Couldn't find a better spot for this. Maybe stash in `core.common` or `core.utils`? Could also just have it as a free function in this module.

cc [USER_REDACTED] if you have any throughts.

this seems like a reasonable spot

Yea, no objection. Just thinking about what the error should say that we pass through to the user. Suggestions welcome, but I should have something by tomorrow.

Added.

Error message looks good (other option would also be to just return False for this function and thus not fallback, but that's probably more confusing for the user).

My initial idea was to put essentially the whole content of the function between the try and expect, in case some attribute of `_parser` that we access there would change or disappear, but again not sure how likely that would actually be

If we were to return False then in the lookaround case we'd raise a not so straightforward error message. This is probably a less likely case so most users would not notice any issue. But then it would end up failing in end up in those sporadic cases which I don't love. Still, I would be okay here if you think it's the better option.

To to cover all cases, I'm more opposed to returning True.

Regarding wrapping the function call, _parser.parse can fail and I don't feel confident predicting all the ways it can fail. So especially if we're returning False on exception, I'm quite negative here.


API: to_datetime with round-floats give requested unit

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.
- [ ] If I used AI to develop this pull request, I prompted it to follow `AGENTS.md`.

Context: https://github.com/[USER_REDACTED]ndas-dev/pandas/issues/63275

@@ -389,6 +389,18 @@ cpdef array_to_datetime(

                 if val != val or val == NPY_NAT:
                     iresult[i] = NPY_NAT
+                elif val.is_integer():
+                    # If we have a round float, treat it like an integer
+                    item_reso = int_reso
+                    state.update_creso(item_reso)
+                    if infer_reso:
+                        creso = state.creso
+
+                    iresult[i] = cast_from_unit(
+                        <int64_t>val, unit_for_numerics, out_reso=creso
+                    )
+
+                    state.found_other = True
                 else:
                     item_reso = NPY_FR_ns
                     state.update_creso(item_reso)

@@ -386,7 +386,7 @@ cdef _TSObject convert_to_tsobject(object ts, tzinfo tz, str unit,
                 obj.value = tz_localize_to_utc_single(
                     obj.value, tz, ambiguous="raise", nonexistent=None, creso=reso
                 )
-    elif is_integer_object(ts):
+    elif is_integer_object(ts) or (is_float_object(ts) and ts.is_integer()):
         try:
             ts = <int64_t>ts
         except OverflowError:

@@ -513,6 +513,17 @@ def _to_datetime_with_unit(arg, unit, name, utc: bool, errors: str) -> Index:
             tz_parsed = None

         elif arg.dtype.kind == "f":
+            with np.errstate(invalid="ignore"):
+                int_values = arg.astype(np.int64)
+            mask = np.isnan(arg)
+            if (mask | (arg == int_values)).all():
+                # With all-round-or-NaN entries, we give the requested unit
+                #  back like with integers
+                result = _to_datetime_with_unit(
+                    int_values, unit=unit, name=name, utc=utc, errors=errors
+                )
+                result._data[mask] = NaT
+                return result
             with np.errstate(over="raise"):
                 try:
                     arr = cast_from_unit_vectorized(arg, unit=unit)

@@ -52,7 +52,7 @@ def test_construct_from_int_float_with_unit_out_of_bound_raises(self, typ):
         # GH#50870  make sure we get an OutOfBoundsDatetime instead of OverflowError
         val = typ(150000000000000)

-        msg = f"cannot convert input {val} with the unit 'D'"
+        msg = f"cannot convert input {int(val)} with the unit 'D'"
         with pytest.raises(OutOfBoundsDatetime, match=msg):
             Timestamp(val, unit="D")

@@ -1782,7 +1782,7 @@ class TestToDatetimeUnit:
     def test_to_datetime_month_or_year_unit_int(self, cache, unit, item, request):
         # GH#50870 Note we have separate tests that pd.Timestamp gets these right
         ts = Timestamp(item, unit=unit)
-        dtype = "M8[ns]" if isinstance(item, float) else "M8[s]"
+        dtype = "M8[s]"
         expected = DatetimeIndex([ts], dtype=dtype)

         result = to_datetime([item], unit=unit, cache=cache)
@@ -1797,7 +1797,7 @@ def test_to_datetime_month_or_year_unit_int(self, cache, unit, item, request):
         # with a nan!
         result = to_datetime(np.array([item, np.nan]), unit=unit, cache=cache)
         assert result.isna()[1]
-        tm.assert_index_equal(result[:1], expected.astype("M8[ns]"))
+        tm.assert_index_equal(result[:1], expected.astype("M8[s]"))

     [USER_REDACTED].mark.parametrize("unit", ["Y", "M"])
     def test_to_datetime_month_or_year_unit_non_round_float(self, cache, unit):
@@ -1824,9 +1824,10 @@ def test_to_datetime_month_or_year_unit_non_round_float(self, cache, unit):
         expected = to_datetime([NaT])
         tm.assert_index_equal(res, expected)

-        # round floats are OK
+        # round floats are OK; treated like integers to give
+        #  closest-to-supported unit
         res = to_datetime([1.0], unit=unit)
-        expected = to_datetime([1], unit=unit).as_unit("ns")
+        expected = to_datetime([1], unit=unit).as_unit("s")
         tm.assert_index_equal(res, expected)

     def test_unit(self, cache):
@@ -1842,7 +1843,7 @@ def test_unit_array_mixed_nans(self, cache):
         result = to_datetime(values, unit="D", errors="coerce", cache=cache)
         expected = DatetimeIndex(
             ["NaT", "1970-01-02", "1970-01-02", "NaT", "NaT", "NaT", "NaT", "NaT"],
-            dtype="M8[ns]",
+            dtype="M8[s]",
         )
         tm.assert_index_equal(result, expected)

@@ -1951,7 +1952,7 @@ def test_to_datetime_unit(self, dtype):
         epoch = 1370745748
         ser = Series([epoch + t for t in range(20)]).astype(dtype)
         result = to_datetime(ser, unit="s")
-        unit = "s" if dtype is int else "ns"
+        unit = "s"
         expected = Series(
             [
                 Timestamp("2013-06-09 02:42:28") + timedelta(seconds=t)
@@ -1968,7 +1969,7 @@ def test_to_datetime_unit_with_nulls(self, null):
         result = to_datetime(ser, unit="s")
         # With np.nan, the list gets cast to a float64 array, which always
         #  gets ns unit.
-        unit = "ns" if null is np.nan else "s"
+        unit = "s"
         expected = Series(
             [Timestamp("2013-06-09 02:42:28") + timedelta(seconds=t) for t in range(20)]
             + [NaT],
@@ -2568,7 +2569,7 @@ def test_to_datetime_float_with_nans_floating_point_error(self):
         ser = Series([np.nan] * 1000 + [1712219033.0], dtype=np.float64)
         result = to_datetime(ser, unit="s", errors="coerce")
         expected = Series(
-            [NaT] * 1000 + [Timestamp("2024-04-04 08:23:53")], dtype="datetime64[ns]"
+            [NaT] * 1000 + [Timestamp("2024-04-04 08:23:53")], dtype="datetime64[s]"
         )
         tm.assert_series_equal(result, expected)

Are we guaranteed to have a numpy object at this point? I see `is_float_object` allows Python floats which doesn't have `is_interger`

> Python floats which doesn't have is_interger

I don't think that's accurate.  Grepping for \.is_integer\(\) I see we use it in a few places

Ah sorry. I see https://docs.python.org/3/library/stdtypes.html#float.is_integer exists now (I didn't see it originally in https://docs.python.org/3/library/functions.html#float)


DOC: Inline docstrings for StringArray methods in string_.py

- Related to issue #62437
- Removed two [USER_REDACTED] decorators in pandas/core/arrays/string_.py
- Extracted the rendered docstrings at runtime and inlined them directly
- Cleaned formatting to match pandas docstring conventions

@@ -32,7 +32,6 @@
 from pandas.compat.numpy import function as nv
 from pandas.errors import Pandas4Warning
 from pandas.util._decorators import (
-    doc,
     set_module,
 )
 from pandas.util._exceptions import find_stack_level
@@ -429,8 +428,24 @@ def _logical_method(self, other, op):
             return op(other, self.astype(bool))
         return NotImplemented

-    [USER_REDACTED](ExtensionArray.tolist)
     def tolist(self) -> list:
+        """
+        Return a list of the value.
+
+        These are each a scalar type, which is a Python scalar
+        (for str, int, float) or pandas scalar
+        (for Timestamp/Timedelta/Interval/Period)
+
+        Returns
+        ----------
+        list
+
+        Examples
+        ----------
+        >>> arr = pd.array(["a", "b", "c"])
+        >>> arr.tolist()
+        ['a', 'b', 'c']
+        """
         if self.ndim > 1:
             return [x.tolist() for x in self]
         return list(self.to_numpy())
@@ -1076,13 +1091,56 @@ def memory_usage(self, deep: bool = False) -> int:
             return result + lib.memory_usage_of_objects(self._ndarray)
         return result

-    [USER_REDACTED](ExtensionArray.searchsorted)
     def searchsorted(
         self,
         value: NumpyValueArrayLike | ExtensionArray,
         side: Literal["left", "right"] = "left",
         sorter: NumpySorter | None = None,
     ) -> npt.NDArray[np.intp] | np.intp:
+        """
+        Find indices where elements should be inserted to maintain order.
+
+        Find the indices into a sorted array `self` (a) such that, if the
+        corresponding elements in `value` were inserted before the indices,
+        the order of `self` would be preserved.
+
+        Assuming that `self` is sorted:
+
+        ======  ================================
+        `side`  returned index `i` satisfies
+        ======  ================================
+        left    ``self[i-1] < value <= self[i]``
+        right   ``self[i-1] <= value < self[i]``
+        ======  ================================
+
+        Parameters
+        ----------
+        value : array-like, list or scalar
+            Value(s) to insert into `self`.
+        side : {'left', 'right'}, optional
+            If 'left', the index of the first suitable location found is given.
+            If 'right', return the last such index.  If there is no suitable
+            index, return either 0 or N (where N is the length of `self`).
+        sorter : 1-D array-like, optional
+            Optional array of integer indices that sort array a into ascending
+            order. They are typically the result of argsort.
+
+        Returns
+        -------
+        array of ints or int
+            If value is array-like, array of insertion points.
+            If value is scalar, a single integer.
+
+        See Also
+        --------
+        numpy.searchsorted : Similar method from NumPy.
+
+        Examples
+        --------
+        >>> arr = pd.array([1, 2, 3, 5])
+        >>> arr.searchsorted([4])
+        array([3])
+        """
         if self._hasna:
             raise ValueError(
                 "searchsorted requires array to be sorted, which is impossible "

Can you change this example to show string data?


BUG: setitem with list of NaT

- [x] closes #63420 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.
- [ ] If I used AI to develop this pull request, I prompted it to follow `AGENTS.md`.

@@ -691,6 +691,14 @@ def _validate_listlike(self, value, allow_object: bool = False):
                     msg = self._validation_error_message(value, True)
                     raise TypeError(msg) from err

+        if isinstance(value, list):
+            value = construct_1d_object_array_from_listlike(value)
+        if isinstance(value, np.ndarray) and value.dtype == object:
+            # We need to call maybe_convert_objects here instead of in pd_array
+            #  so we can specify dtype_if_all_nat.
+            value = lib.maybe_convert_objects(
+                value, convert_non_numeric=True, dtype_if_all_nat=self.dtype
+            )
         # Do type inference if necessary up front (after unpacking
         # NumpyExtensionArray)
         # e.g. we passed PeriodIndex.values and got an ndarray of Periods

@@ -334,7 +334,7 @@ def array(
                 ensure_object(data),
                 convert_non_numeric=True,
                 convert_to_nullable_dtype=True,
-                dtype_if_all_nat=None,
+                dtype_if_all_nat=np.dtype("M8[s]"),
             )
             result = ensure_wrapped_if_datetimelike(result)
             if isinstance(result, np.ndarray):

@@ -134,6 +134,11 @@ def test_dt64_array(dtype_unit):
             np.dtype("datetime64[ns]"),
             DatetimeArray._from_sequence(["2000", "2001"], dtype="M8[ns]"),
         ),
+        (
+            [pd.NaT, pd.NaT],
+            None,
+            DatetimeArray._from_sequence([pd.NaT, pd.NaT], dtype="M8[s]"),
+        ),
         # Datetime (tz-aware)
         (
             ["2000", "2001"],

@@ -301,7 +301,9 @@ def test_searchsorted(self):
         assert result == 10

     [USER_REDACTED].mark.parametrize("box", [None, "index", "series"])
-    def test_searchsorted_castable_strings(self, arr1d, box, string_storage):
+    def test_searchsorted_castable_strings(
+        self, arr1d, box, string_storage, using_infer_string
+    ):
         arr = arr1d
         if box is None:
             pass
@@ -332,13 +334,16 @@ def test_searchsorted_castable_strings(self, arr1d, box, string_storage):
         ):
             arr.searchsorted("foo")

+        msg = re.escape(
+            f"value should be a '{arr1d._scalar_type.__name__}', 'NaT', "
+            "or array of those. Got str array instead."
+        )
+        if not using_infer_string:
+            msg = msg.replace("str", "string")
         with pd.option_context("string_storage", string_storage):
             with pytest.raises(
                 TypeError,
-                match=re.escape(
-                    f"value should be a '{arr1d._scalar_type.__name__}', 'NaT', "
-                    "or array of those. Got string array instead."
-                ),
+                match=msg,
             ):
                 arr.searchsorted([str(arr[1]), "baz"])

@@ -433,6 +438,11 @@ def test_setitem(self):
         expected[:2] = expected[-2:]
         tm.assert_numpy_array_equal(arr.asi8, expected)

+    def test_setitem_list_of_nats(self, arr1d):
+        # GH#63420
+        arr1d[:] = [NaT] * len(arr1d)
+        assert arr1d.isna().all()
+
     [USER_REDACTED].mark.parametrize(
         "box",
         [

Could absorb the `if isinstance(value, list) and len(value) == 0` branch above here as well.

Also do we need to be concerned about any other non-ndarray iterable here e.g. tuple?

we could group them together, but i think we still need separate clauses.

could handle tuples, but i think in general we should move toward being stricter about what we allow, so prefer not to unless it causes problems


DOC: inline docstring in CalamineReader

- [ ] xref  #62437
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.
- [ ] If I used AI to develop this pull request, I prompted it to follow `AGENTS.md`.

Replaced `[USER_REDACTED]` decorator with hardcoded docstrings in `pandas/io/excel/_calamine.py`:
- **CalamineReader.__init__**: Inlined `storage_options` docstring and removed the `[USER_REDACTED]` decorator.
- Removed unused imports (`doc`, `_shared_docs`).

@@ -13,9 +13,6 @@
 )

 from pandas.compat._optional import import_optional_dependency
-from pandas.util._decorators import doc
-
-from pandas.core.shared_docs import _shared_docs

 from pandas.io.excel._base import BaseExcelReader

@@ -37,7 +34,6 @@

 class CalamineReader(BaseExcelReader["CalamineWorkbook"]):
-    [USER_REDACTED](storage_options=_shared_docs["storage_options"])
     def __init__(
         self,
         filepath_or_buffer: FilePath | ReadBuffer[bytes],
@@ -51,7 +47,15 @@ def __init__(
         ----------
         filepath_or_buffer : str, path to be parsed or
             an open readable stream.
-        {storage_options}
+        storage_options : dict, optional
+            Extra options that make sense for a particular storage connection, e.g.
+            host, port, username, password, etc. For HTTP(S) URLs the key-value pairs
+            are forwarded to ``urllib.request.Request`` as header options. For other
+            URLs (e.g. starting with "s3://", and "gcs://") the key-value pairs are
+            forwarded to ``fsspec.open``. Please see ``fsspec`` and ``urllib`` for more
+            details, and for more examples on storage options refer `here
+            <https://pandas.pydata.org/docs/user_guide/io.html?
+            highlight=storage_options#reading-writing-remote-files>`_.
         engine_kwargs : dict, optional
             Arbitrary keyword arguments passed to excel engine.
         """

Can you indent this entire description?

Thanks for the review! I've updated the indentation as requested.


BUG: Arrow-backed duration/timestamp reductions return stdlib types for non-nanosecond units

- [x] closes #63170
- [x] [Tests added and passed] if fixing a bug or adding a new feature
- [x] All [code checks passed].
- [ ] Added [type annotations] to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.
- [ ] If I used AI to develop this pull request, I prompted it to follow `AGENTS.md`.

Previously, reduction operations (`sum`, `min`, `max`, `mean`, `median`) on Arrow-backed duration and timestamp arrays returned inconsistent scalar types depending on unit resolution. PyArrow's `as_py()` returns `pd.Timedelta/pd.Timestamp` for nanosecond data but stdlib `datetime.timedelta/datetime.datetime` for microsecond, millisecond, and second resolutions.
This change adds explicit wrapping in `_reduce_calc` for duration and timestamp results, using the pattern already established for[`__getitem__`](https://github.com/[USER_REDACTED]ndas-dev/pandas/blob/91514c363604506f447e53380d3aa00520f1037b/pandas/core/arrays/arrow/array.py#L807-L814) and [`__iter__`](https://github.com/[USER_REDACTED]ndas-dev/pandas/blob/91514c363604506f447e53380d3aa00520f1037b/pandas/core/arrays/arrow/array.py#L823-L834). This ensures these reductions consistently return pandas scalar types while preserving the original unit resolution via `.as_unit()`.

@@ -1401,6 +1401,7 @@ ExtensionArray
 - Bug in :meth:`ArrowExtensionArray.factorize` where NA values were dropped when input was dictionary-encoded even when dropna was set to False(:issue:`60567`)
 - Bug in :meth:`NDArrayBackedExtensionArray.take` which produced arrays whose dtypes didn't match their underlying data, when called with integer arrays (:issue:`62448`)
 - Bug in :meth:`api.types.is_datetime64_any_dtype` where a custom :class:`ExtensionDtype` would return ``False`` for array-likes (:issue:`57055`)
+- Bug in Arrow-backed duration and timestamp reductions (e.g. ``sum``, ``min``, ``max``, ``median``, ``mean``) incorrectly returning stdlib ``datetime`` objects instead of :class:`Timedelta` and :class:`Timestamp` for resolutions other than nanoseconds (:issue:`63170`)
 - Bug in comparison between object with :class:`ArrowDtype` and incompatible-dtyped (e.g. string vs bool) incorrectly raising instead of returning all-``False`` (for ``==``) or all-``True`` (for ``!=``) (:issue:`59505`)
 - Bug in constructing pandas data structures when passing into ``dtype`` a string of the type followed by ``[pyarrow]`` while PyArrow is not installed would raise ``NameError`` rather than ``ImportError`` (:issue:`57928`)
 - Bug in various :class:`DataFrame` reductions for pyarrow temporal dtypes returning incorrect dtype when result was null (:issue:`59234`)

@@ -2152,7 +2152,13 @@ def _reduce_calc(
         if pc.is_null(pa_result).as_py():
             return self.dtype.na_value
         elif isinstance(pa_result, pa.Scalar):
-            return pa_result.as_py()
+            result = pa_result.as_py()
+            pa_type = pa_result.type
+            if pa.types.is_duration(pa_type) and pa_type.unit != "ns":
+                return Timedelta(result).as_unit(pa_type.unit)
+            elif pa.types.is_timestamp(pa_type) and pa_type.unit != "ns":
+                return Timestamp(result).as_unit(pa_type.unit)
+            return result
         else:
             return pa_result

@@ -3817,3 +3817,27 @@ def test_ufunc_retains_missing():

     expected = pd.Series([np.sin(0.1), pd.NA], dtype="float64[pyarrow]")
     tm.assert_series_equal(result, expected)
+
+
+[USER_REDACTED].mark.parametrize("method", ["sum", "min", "max", "mean", "median"])
+def test_duration_reduction_consistency(unit, method):
+    # GH#63170
+    dtype = f"duration[{unit}][pyarrow]"
+    ser = pd.Series([timedelta(seconds=1), timedelta(seconds=2)], dtype=dtype)
+    result = getattr(ser, method)()
+    assert isinstance(result, pd.Timedelta), (
+        f"{method} for {unit} returned {type(result)}"
+    )
+    assert result.unit == unit
+
+
+[USER_REDACTED].mark.parametrize("method", ["min", "max", "median"])
+def test_timestamp_reduction_consistency(unit, method):
+    # GH#63170
+    dtype = f"timestamp[{unit}][pyarrow]"
+    ser = pd.Series([datetime(2024, 1, 1), datetime(2024, 1, 3)], dtype=dtype)
+    result = getattr(ser, method)()
+    assert isinstance(result, pd.Timestamp), (
+        f"{method} for {unit} returned {type(result)}"
+    )
+    assert result.unit == unit

You can use the `unit` fixture

Can you use `pytest.mark.parametrize` over the reduction methods (same as below)?


DOC: Inline docstrings of a batch of functions in `pandas/core/generic.py`

- [ ] xref #62437 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.
- [ ] If I used AI to develop this pull request, I prompted it to follow `AGENTS.md`.

@@ -188,8 +188,8 @@
 from pandas.io.formats.printing import pprint_thing

 if TYPE_CHECKING:
-    from collections.abc import Callable
     from collections.abc import (
+        Callable,
         Hashable,
         Iterator,
         Mapping,
@@ -208,7 +208,6 @@
     from pandas.core.indexers.objects import BaseIndexer
     from pandas.core.resample import Resampler

-import textwrap

 # goal is to be able to define the docs close to function, while still being
 # able to share
@@ -767,10 +766,9 @@ def _set_axis(self, axis: AxisInt, labels: AnyArrayLike | list) -> None:
         self._mgr.set_axis(axis, labels)

     [USER_REDACTED]
-    [USER_REDACTED](klass=_shared_doc_kwargs["klass"])
     def droplevel(self, level: IndexLabel, axis: Axis = 0) -> Self:
         """
-        Return {klass} with requested index / column level(s) removed.
+        [NAME_REDACTED]/DataFrame with requested index / column level(s) removed.

         Parameters
         ----------
@@ -789,8 +787,8 @@ def droplevel(self, level: IndexLabel, axis: Axis = 0) -> Self:

         Returns
         -------
-        {klass}
-            {klass} with requested index / column level(s) removed.
+        Series/DataFrame
+            Series/DataFrame with requested index / column level(s) removed.

         See Also
         --------
@@ -2150,19 +2148,6 @@ def _repr_data_resource_(self):
     # I/O Methods

     [USER_REDACTED]
-    [USER_REDACTED](
-        klass="object",
-        storage_options=_shared_docs["storage_options"],
-        storage_options_versionadded="1.2.0",
-        encoding_parameter="",
-        verbose_parameter="",
-        extra_parameters=textwrap.dedent(
-            """\
-        engine_kwargs : dict, optional
-            Arbitrary keyword arguments passed to excel engine.
-        """
-        ),
-    )
     def to_excel(
         self,
         excel_writer: FilePath | WriteExcelBuffer | ExcelWriter,
@@ -2185,9 +2170,9 @@ def to_excel(
         autofilter: bool = False,
     ) -> None:
         """
-        Write {klass} to an Excel sheet.
+        Write object to an Excel sheet.

-        To write a single {klass} to an Excel .xlsx file it is only necessary to
+        To write a single object to an Excel .xlsx file it is only necessary to
         specify a target file name. To write to multiple sheets it is necessary to
         create an `ExcelWriter` object with a target file name, and specify a sheet
         in the file to write to.
@@ -2227,22 +2212,27 @@ def to_excel(
             Write engine to use, 'openpyxl' or 'xlsxwriter'. You can also set this
             via the options ``io.excel.xlsx.writer`` or
             ``io.excel.xlsm.writer``.
-
         merge_cells : bool or 'columns', default False
             If True, write MultiIndex index and columns as merged cells.
             If 'columns', merge MultiIndex column cells only.
-        {encoding_parameter}
         inf_rep : str, default 'inf'
             Representation for infinity (there is no native representation for
             infinity in Excel).
-        {verbose_parameter}
         freeze_panes : tuple of int (length 2), optional
             Specifies the one-based bottommost row and rightmost column that
             is to be frozen.
-        {storage_options}
+        storage_options : dict, optional
+            Extra options that make sense for a particular storage connection, e.g.
+            host, port, username, password, etc. For HTTP(S) URLs the key-value pairs
+            are forwarded to ``urllib.request.Request`` as header options. For other
+            URLs (e.g. starting with "s3://", and "gcs://") the key-value pairs are
+            forwarded to ``fsspec.open``. Please see ``fsspec`` and ``urllib`` for more
+            details, and for more examples on storage options refer `here
+            <https://pandas.pydata.org/docs/user_guide/io.html?
+            highlight=storage_options#reading-writing-remote-files>`_.

-            .. versionadded:: {storage_options_versionadded}
-        {extra_parameters}
+        engine_kwargs : dict, optional
+            Arbitrary keyword arguments passed to excel engine.
         autofilter : bool, default False
             If True, add automatic filters to all columns.

@@ -2332,10 +2322,6 @@ def to_excel(
         )

     [USER_REDACTED]
-    [USER_REDACTED](
-        storage_options=_shared_docs["storage_options"],
-        compression_options=_shared_docs["compression_options"] % "path_or_buf",
-    )
     def to_json(
         self,
         path_or_buf: FilePath | WriteBuffer[bytes] | WriteBuffer[str] | None = None,
@@ -2420,7 +2406,23 @@ def to_json(
             If 'orient' is 'records' write out line-delimited json format. Will
             throw ValueError if incorrect 'orient' since others are not
             list-like.
-        {compression_options}
+
+        compression : str or dict, default 'infer'
+            For on-the-fly compression of the output data. If 'infer' and
+            'path_or_buf' is path-like, then detect compression from the following
+            extensions: '.gz',
+            '.bz2', '.zip', '.xz', '.zst', '.tar', '.tar.gz', '.tar.xz' or '.tar.bz2'
+            (otherwise no compression).
+            Set to ``None`` for no compression.
+            Can also be a dict with key ``'method'`` set to one of
+            {``'zip'``, ``'gzip'``, ``'bz2'``, ``'zstd'``, ``'xz'``, ``'tar'``} and
+            other key-value pairs are forwarded to
+            ``zipfile.ZipFile``, ``gzip.GzipFile``,
+            ``bz2.BZ2File``, ``zstandard.ZstdCompressor``, ``lzma.LZMAFile`` or
+            ``tarfile.TarFile``, respectively.
+            As an example, the following could be passed for faster compression and
+            to create a reproducible gzip archive:
+            ``compression={'method': 'gzip', 'compresslevel': 1, 'mtime': 1}``.

         index : bool or None, default None
             The index is only used when 'orient' is 'split', 'index', 'column',
@@ -2431,7 +2433,15 @@ def to_json(
         indent : int, optional
            Length of whitespace used to indent each record.

-        {storage_options}
+        storage_options : dict, optional
+            Extra options that make sense for a particular storage connection, e.g.
+            host, port, username, password, etc. For HTTP(S) URLs the key-value pairs
+            are forwarded to ``urllib.request.Request`` as header options. For other
+            URLs (e.g. starting with "s3://", and "gcs://") the key-value pairs are
+            forwarded to ``fsspec.open``. Please see ``fsspec`` and ``urllib`` for more
+            details, and for more examples on storage options refer `here
+            <https://pandas.pydata.org/docs/user_guide/io.html?
+            highlight=storage_options#reading-writing-remote-files>`_.

         mode : str, default 'w' (writing)
             Specify the IO mode for output when supplying a path_or_buf.
@@ -3054,10 +3064,6 @@ def to_sql(
         )

     [USER_REDACTED]
-    [USER_REDACTED](
-        storage_options=_shared_docs["storage_options"],
-        compression_options=_shared_docs["compression_options"] % "path",
-    )
     def to_pickle(
         self,
         path: FilePath | WriteBuffer[bytes],
@@ -3075,7 +3081,24 @@ def to_pickle(
             String, path object (implementing ``os.PathLike[str]``), or file-like
             object implementing a binary ``write()`` function. File path where
             the pickled object will be stored.
-        {compression_options}
+
+        compression : str or dict, default 'infer'
+            For on-the-fly compression of the output data. If 'infer' and
+            'path_or_buf' is path-like, then detect compression from the following
+            extensions: '.gz',
+            '.bz2', '.zip', '.xz', '.zst', '.tar', '.tar.gz', '.tar.xz' or '.tar.bz2'
+            (otherwise no compression).
+            Set to ``None`` for no compression.
+            Can also be a dict with key ``'method'`` set to one of
+            {``'zip'``, ``'gzip'``, ``'bz2'``, ``'zstd'``, ``'xz'``, ``'tar'``} and
+            other key-value pairs are forwarded to
+            ``zipfile.ZipFile``, ``gzip.GzipFile``,
+            ``bz2.BZ2File``, ``zstandard.ZstdCompressor``, ``lzma.LZMAFile`` or
+            ``tarfile.TarFile``, respectively.
+            As an example, the following could be passed for faster compression and
+            to create a reproducible gzip archive:
+            ``compression={'method': 'gzip', 'compresslevel': 1, 'mtime': 1}``.
+
         protocol : int
             Int which indicates which protocol should be used by [NAME_REDACTED],
             default HIGHEST_PROTOCOL (see [1]_ paragraph 12.1.2). The possible
@@ -3084,7 +3107,15 @@ def to_pickle(

             .. [1] https://docs.python.org/3/library/pickle.html.

-        {storage_options}
+        storage_options : dict, optional
+            Extra options that make sense for a particular storage connection, e.g.
+            host, port, username, password, etc. For HTTP(S) URLs the key-value pairs
+            are forwarded to ``urllib.request.Request`` as header options. For other
+            URLs (e.g. starting with "s3://", and "gcs://") the key-value pairs are
+            forwarded to ``fsspec.open``. Please see ``fsspec`` and ``urllib`` for more
+            details, and for more examples on storage options refer `here
+            <https://pandas.pydata.org/docs/user_guide/io.html?
+            highlight=storage_options#reading-writing-remote-files>`_.

         See Also
         --------
@@ -3763,10 +3794,6 @@ def to_csv(
     ) -> None: ...

     [USER_REDACTED]
-    [USER_REDACTED](
-        storage_options=_shared_docs["storage_options"],
-        compression_options=_shared_docs["compression_options"] % "path_or_buf",
-    )
     def to_csv(
         self,
         path_or_buf: FilePath | WriteBuffer[bytes] | WriteBuffer[str] | None = None,
@@ -3835,7 +3862,23 @@ def to_csv(
             A string representing the encoding to use in the output file,
             defaults to 'utf-8'. `encoding` is not supported if `path_or_buf`
             is a non-binary file object.
-        {compression_options}
+
+        compression : str or dict, default 'infer'
+            For on-the-fly compression of the output data. If 'infer' and
+            'path_or_buf' is path-like, then detect compression from the following
+            extensions: '.gz',
+            '.bz2', '.zip', '.xz', '.zst', '.tar', '.tar.gz', '.tar.xz' or '.tar.bz2'
+            (otherwise no compression).
+            Set to ``None`` for no compression.
+            Can also be a dict with key ``'method'`` set to one of
+            {``'zip'``, ``'gzip'``, ``'bz2'``, ``'zstd'``, ``'xz'``, ``'tar'``} and
+            other key-value pairs are forwarded to
+            ``zipfile.ZipFile``, ``gzip.GzipFile``,
+            ``bz2.BZ2File``, ``zstandard.ZstdCompressor``, ``lzma.LZMAFile`` or
+            ``tarfile.TarFile``, respectively.
+            As an example, the following could be passed for faster compression and
+            to create a reproducible gzip archive:
+            ``compression={'method': 'gzip', 'compresslevel': 1, 'mtime': 1}``.

                May be a dict with key 'method' as compression mode
                and other entries as additional compression options if
@@ -3870,7 +3913,15 @@ def to_csv(
             See the errors argument for :func:`open` for a full list
             of options.

-        {storage_options}
+        storage_options : dict, optional
+            Extra options that make sense for a particular storage connection, e.g.
+            host, port, username, password, etc. For HTTP(S) URLs the key-value pairs
+            are forwarded to ``urllib.request.Request`` as header options. For other
+            URLs (e.g. starting with "s3://", and "gcs://") the key-value pairs are
+            forwarded to ``fsspec.open``. Please see ``fsspec`` and ``urllib`` for more
+            details, and for more examples on storage options refer `here
+            <https://pandas.pydata.org/docs/user_guide/io.html?
+            highlight=storage_options#reading-writing-remote-files>`_.

         Returns
         -------
@@ -5148,10 +5199,6 @@ def sort_index(
         else:
             return result.__finalize__(self, method="sort_index")

-    [USER_REDACTED](
-        klass=_shared_doc_kwargs["klass"],
-        optional_reindex="",
-    )
     def reindex(
         self,
         labels=None,
@@ -5167,15 +5214,14 @@ def reindex(
         tolerance=None,
     ) -> Self:
         """
-        Conform {klass} to new index with optional filling logic.
+        [NAME_REDACTED]/DataFrame to new index with optional filling logic.

         Places NA/NaN in locations having no value in the previous index. A new object
         is produced unless the new index is equivalent to the current one and
         ``copy=False``.

         Parameters
         ----------
-        {optional_reindex}
         method : {{None, 'backfill'/'bfill', 'pad'/'ffill', 'nearest'}}
             Method to use for filling holes in reindexed DataFrame.
             Please note: this is only applicable to DataFrames/Series with a
@@ -5221,8 +5267,8 @@ def reindex(

         Returns
         -------
-        {klass}
-            {klass} with changed index.
+        Series/DataFrame
+            Series/DataFrame with changed index.

         See Also
         --------
@@ -5318,7 +5364,7 @@ def reindex(

         >>> date_index = pd.date_range("1/1/2010", periods=6, freq="D")
         >>> df2 = pd.DataFrame(
-        ...     {{"prices": [100, 101, np.nan, 100, 89, 88]}}, index=date_index
+        ...     {"prices": [100, 101, np.nan, 100, 89, 88]}, index=date_index
         ... )
         >>> df2
                     prices
@@ -5978,7 +6024,6 @@ def pipe(
     ) -> T: ...

     [USER_REDACTED]
-    [USER_REDACTED](klass=_shared_doc_kwargs["klass"])
     def pipe(
         self,
         func: Callable[Concatenate[Self, P], T] | tuple[Callable[..., T], str],
@@ -5991,11 +6036,11 @@ def pipe(
         Parameters
         ----------
         func : function
-            Function to apply to the {klass}.
+            Function to apply to the Series/DataFrame.
             ``args``, and ``kwargs`` are passed into ``func``.
             Alternatively a ``(callable, data_keyword)`` tuple where
             ``data_keyword`` is a string indicating the keyword of
-            ``callable`` that expects the {klass}.
+            ``callable`` that expects the Series/DataFrame.
         *args : iterable, optional
             Positional arguments passed into ``func``.
         **kwargs : mapping, optional
@@ -6918,10 +6963,6 @@ def _pad_or_backfill(
             return result.__finalize__(self, method="fillna")

     [USER_REDACTED]
-    [USER_REDACTED](
-        klass=_shared_doc_kwargs["klass"],
-        axes_single_arg=_shared_doc_kwargs["axes_single_arg"],
-    )
     def fillna(
         self,
         value: Hashable | Mapping | Series | DataFrame,
@@ -6941,7 +6982,7 @@ def fillna(
             each index (for a Series) or column (for a DataFrame).  Values not
             in the dict/Series/DataFrame will not be filled. This value cannot
             be a list.
-        axis : {axes_single_arg}
+        axis : {0 or 'index'} for Series, {0 or 'index', 1 or 'columns'} for [NAME_REDACTED] along which to fill missing values. For `Series`
             this parameter is unused and defaults to 0.
         inplace : bool, default False
@@ -6954,7 +6995,7 @@ def fillna(

         Returns
         -------
-        {klass}
+        Series/[NAME_REDACTED] with missing values filled.

         See Also
@@ -7001,7 +7042,7 @@ def fillna(
         Replace all NaN elements in column 'A', 'B', 'C', and 'D', with 0, 1,
         2, and 3 respectively.

-        >>> values = {{"A": 0, "B": 1, "C": 2, "D": 3}}
+        >>> values = {"A": 0, "B": 1, "C": 2, "D": 3}
         >>> df.fillna(value=values)
              A    B    C    D
         0  0.0  2.0  2.0  0.0
@@ -7164,10 +7205,6 @@ def fillna(
             return result.__finalize__(self, method="fillna")

     [USER_REDACTED]
-    [USER_REDACTED](
-        klass=_shared_doc_kwargs["klass"],
-        axes_single_arg=_shared_doc_kwargs["axes_single_arg"],
-    )
     def ffill(
         self,
         *,
@@ -7181,7 +7218,7 @@ def ffill(

         Parameters
         ----------
-        axis : {axes_single_arg}
+        axis : {0 or 'index'} for Series, {0 or 'index', 1 or 'columns'} for [NAME_REDACTED] along which to fill missing values. For `Series`
             this parameter is unused and defaults to 0.
         inplace : bool, default False
@@ -7208,7 +7245,7 @@ def ffill(

         Returns
         -------
-        {klass}
+        Series/[NAME_REDACTED] with missing values filled.

         See Also
@@ -7270,10 +7307,6 @@ def ffill(
         )

     [USER_REDACTED]
-    [USER_REDACTED](
-        klass=_shared_doc_kwargs["klass"],
-        axes_single_arg=_shared_doc_kwargs["axes_single_arg"],
-    )
     def bfill(
         self,
         *,
@@ -7287,7 +7320,7 @@ def bfill(

         Parameters
         ----------
-        axis : {axes_single_arg}
+        axis : {0 or 'index'} for Series, {0 or 'index', 1 or 'columns'} for [NAME_REDACTED] along which to fill missing values. For `Series`
             this parameter is unused and defaults to 0.
         inplace : bool, default False
@@ -7314,7 +7347,7 @@ def bfill(

         Returns
         -------
-        {klass}
+        Series/[NAME_REDACTED] with missing values filled.

         See Also
@@ -7342,7 +7375,7 @@ def bfill(

         With DataFrame:

-        >>> df = pd.DataFrame({{"A": [1, None, None, 4], "B": [None, 5, None, 7]}})
+        >>> df = pd.DataFrame({"A": [1, None, None, 4], "B": [None, 5, None, 7]})
         >>> df
               A     B
         0   1.0	  NaN

```suggestion
```


TST Refactor out helper in `pos_label` display curve tests

#### [NAME_REDACTED]/PRs
Related to #33218 and https://github.com/[USER_REDACTED]n/scikit-learn/pull/33217
Part of PRs to move test related changes out of #30508 and into separate PR

#### What does this implement/fix? Explain your changes.
Takes the common part of `test_plot_roc_curve_pos_label` (which is shared with `test_plot_precision_recall_pos_label`) out to a helper function in `test_common_curve_display.py`, to reduce duplication.

#### AI usage disclosure
<!--
If AI tools were involved in creating this PR, please check all boxes that apply
below and make sure that you adhere to our [NAME_REDACTED]:
https://scikit-learn.org/dev/developers/contributing.html#automated-contributions-policy
-->
I used AI assistance for:
- [ ] Code generation (e.g., when writing an implementation or fixing a bug)
- [ ] Test/benchmark generation
- [ ] Documentation (including examples)
- [ ] Research and understanding

#### Any other comments?

cc [USER_REDACTED] [USER_REDACTED] :pray:

@@ -4,7 +4,7 @@
 from sklearn.base import BaseEstimator, ClassifierMixin, clone
 from sklearn.calibration import CalibrationDisplay
 from sklearn.compose import make_column_transformer
-from sklearn.datasets import load_iris
+from sklearn.datasets import load_breast_cancer, load_iris
 from sklearn.exceptions import NotFittedError
 from sklearn.linear_model import LogisticRegression
 from sklearn.metrics import (
@@ -14,9 +14,11 @@
     PredictionErrorDisplay,
     RocCurveDisplay,
 )
+from sklearn.model_selection import cross_validate, train_test_split
 from sklearn.pipeline import make_pipeline
 from sklearn.preprocessing import StandardScaler
 from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor
+from sklearn.utils import shuffle

 [USER_REDACTED].fixture(scope="module")
@@ -30,6 +32,99 @@ def data_binary(data):
     return X[y < 2], y[y < 2]

+def _check_pos_label_statistics(
+    display_class, response_method, constructor_name, check_metric
+):
+    """Test switching `pos_label` gives correct statistics, using imbalanced data."""
+    X, y = load_breast_cancer(return_X_y=True)
+    # create highly imbalanced classes
+    idx_positive = np.flatnonzero(y == 1)
+    idx_negative = np.flatnonzero(y == 0)
+    idx_selected = np.hstack([idx_negative, idx_positive[:25]])
+    X, y = X[idx_selected], y[idx_selected]
+    X, y = shuffle(X, y, random_state=42)
+    # only use 2 features to make the problem even harder
+    X = X[:, :2]
+    y = np.array(["cancer" if c == 1 else "not cancer" for c in y], dtype=object)
+    X_train, X_test, y_train, y_test = train_test_split(
+        X,
+        y,
+        stratify=y,
+        random_state=0,
+    )
+
+    classifier = LogisticRegression()
+    classifier.fit(X_train, y_train)
+    cv_results = cross_validate(
+        LogisticRegression(), X, y, cv=3, return_estimator=True, return_indices=True
+    )
+
+    # Sanity check to be sure the positive class is `classes_[0]`.
+    # Class imbalance ensures a large difference in prediction values between classes,
+    # allowing us to catch errors when we switch `pos_label`.
+    assert classifier.classes_.tolist() == ["cancer", "not cancer"]
+
+    y_score = getattr(classifier, response_method)(X_test)
+    # we select the corresponding probability columns or reverse the decision
+    # function otherwise
+    y_score_cancer = -1 * y_score if y_score.ndim == 1 else y_score[:, 0]
+    y_score_not_cancer = y_score if y_score.ndim == 1 else y_score[:, 1]
+
+    pos_label = "cancer"
+    y_score = y_score_cancer
+    if constructor_name == "from_estimator":
+        display = display_class.from_estimator(
+            classifier,
+            X_test,
+            y_test,
+            pos_label=pos_label,
+            response_method=response_method,
+        )
+    elif constructor_name == "from_predictions":
+        display = display_class.from_predictions(
+            y_test,
+            y_score,
+            pos_label=pos_label,
+        )
+    else:
+        display = display_class.from_cv_results(
+            cv_results,
+            X,
+            y,
+            response_method=response_method,
+            pos_label=pos_label,
+        )
+
+    check_metric(display, constructor_name, pos_label)
+
+    pos_label = "not cancer"
+    y_score = y_score_not_cancer
+    if constructor_name == "from_estimator":
+        display = display_class.from_estimator(
+            classifier,
+            X_test,
+            y_test,
+            response_method=response_method,
+            pos_label=pos_label,
+        )
+    elif constructor_name == "from_predictions":
+        display = display_class.from_predictions(
+            y_test,
+            y_score,
+            pos_label=pos_label,
+        )
+    else:
+        display = display_class.from_cv_results(
+            cv_results,
+            X,
+            y,
+            response_method=response_method,
+            pos_label=pos_label,
+        )
+
+    check_metric(display, constructor_name, pos_label)
+
+
 [USER_REDACTED].mark.parametrize(
     "Display",
     [CalibrationDisplay, DetCurveDisplay, PrecisionRecallDisplay, RocCurveDisplay],

@@ -7,14 +7,17 @@

 from sklearn import clone
 from sklearn.compose import make_column_transformer
-from sklearn.datasets import load_breast_cancer, make_classification
+from sklearn.datasets import make_classification
 from sklearn.exceptions import NotFittedError, UndefinedMetricWarning
 from sklearn.linear_model import LogisticRegression
 from sklearn.metrics import RocCurveDisplay, auc, roc_curve
-from sklearn.model_selection import cross_validate, train_test_split
+from sklearn.metrics._plot.tests.test_common_curve_display import (
+    _check_pos_label_statistics,
+)
+from sklearn.model_selection import cross_validate
 from sklearn.pipeline import make_pipeline
 from sklearn.preprocessing import StandardScaler
-from sklearn.utils import _safe_indexing, shuffle
+from sklearn.utils import _safe_indexing
 from sklearn.utils._response import _get_response_values_binary

@@ -822,112 +825,27 @@ def test_roc_curve_display_default_labels(
         assert disp.line_[idx].get_label() == expected_label

-def _check_auc(display, constructor_name):
-    roc_auc_limit = 0.95679
-    roc_auc_limit_multi = [0.97007, 0.985915, 0.980952]
-
-    if constructor_name == "from_cv_results":
-        for idx, roc_auc in enumerate(display.roc_auc):
-            assert roc_auc == pytest.approx(roc_auc_limit_multi[idx])
-    else:
-        assert display.roc_auc == pytest.approx(roc_auc_limit)
-        assert trapezoid(display.tpr, display.fpr) == pytest.approx(roc_auc_limit)
-
-
 [USER_REDACTED].mark.parametrize("response_method", ["predict_proba", "decision_function"])
 [USER_REDACTED].mark.parametrize(
     "constructor_name", ["from_estimator", "from_predictions", "from_cv_results"]
 )
 def test_plot_roc_curve_pos_label(pyplot, response_method, constructor_name):
-    # check that we can provide the positive label and display the proper
-    # statistics
-    X, y = load_breast_cancer(return_X_y=True)
-    # create a highly imbalanced version of the breast cancer dataset
-    idx_positive = np.flatnonzero(y == 1)
-    idx_negative = np.flatnonzero(y == 0)
-    idx_selected = np.hstack([idx_negative, idx_positive[:25]])
-    X, y = X[idx_selected], y[idx_selected]
-    X, y = shuffle(X, y, random_state=42)
-    # only use 2 features to make the problem even harder
-    X = X[:, :2]
-    y = np.array(["cancer" if c == 1 else "not cancer" for c in y], dtype=object)
-    X_train, X_test, y_train, y_test = train_test_split(
-        X,
-        y,
-        stratify=y,
-        random_state=0,
-    )
-
-    classifier = LogisticRegression()
-    classifier.fit(X_train, y_train)
-    cv_results = cross_validate(
-        LogisticRegression(), X, y, cv=3, return_estimator=True, return_indices=True
-    )
-
-    # Sanity check to be sure the positive class is `classes_[0]`
-    # Class imbalance ensures a large difference in prediction values between classes,
-    # allowing us to catch errors when we switch `pos_label`
-    assert classifier.classes_.tolist() == ["cancer", "not cancer"]
+    """Test switching `pos_label` gives correct statistics, using imbalanced data."""

-    y_score = getattr(classifier, response_method)(X_test)
-    # we select the corresponding probability columns or reverse the decision
-    # function otherwise
-    y_score_cancer = -1 * y_score if y_score.ndim == 1 else y_score[:, 0]
-    y_score_not_cancer = y_score if y_score.ndim == 1 else y_score[:, 1]
+    def _check_auc(display, constructor_name, pos_label):
+        roc_auc_limit = 0.95679
+        roc_auc_limit_multi = [0.97007, 0.985915, 0.980952]

-    pos_label = "cancer"
-    y_score = y_score_cancer
-    if constructor_name == "from_estimator":
-        display = RocCurveDisplay.from_estimator(
-            classifier,
-            X_test,
-            y_test,
-            pos_label=pos_label,
-            response_method=response_method,
-        )
-    elif constructor_name == "from_predictions":
-        display = RocCurveDisplay.from_predictions(
-            y_test,
-            y_score,
-            pos_label=pos_label,
-        )
-    else:
-        display = RocCurveDisplay.from_cv_results(
-            cv_results,
-            X,
-            y,
-            response_method=response_method,
-            pos_label=pos_label,
-        )
-
-    _check_auc(display, constructor_name)
-
-    pos_label = "not cancer"
-    y_score = y_score_not_cancer
-    if constructor_name == "from_estimator":
-        display = RocCurveDisplay.from_estimator(
-            classifier,
-            X_test,
-            y_test,
-            response_method=response_method,
-            pos_label=pos_label,
-        )
-    elif constructor_name == "from_predictions":
-        display = RocCurveDisplay.from_predictions(
-            y_test,
-            y_score,
-            pos_label=pos_label,
-        )
-    else:
-        display = RocCurveDisplay.from_cv_results(
-            cv_results,
-            X,
-            y,
-            response_method=response_method,
-            pos_label=pos_label,
-        )
+        if constructor_name == "from_cv_results":
+            for idx, roc_auc in enumerate(display.roc_auc):
+                assert roc_auc == pytest.approx(roc_auc_limit_multi[idx])
+        else:
+            assert display.roc_auc == pytest.approx(roc_auc_limit)
+            assert trapezoid(display.tpr, display.fpr) == pytest.approx(roc_auc_limit)

-    _check_auc(display, constructor_name)
+    _check_pos_label_statistics(
+        RocCurveDisplay, response_method, constructor_name, _check_auc
+    )

 # TODO(1.9): remove

LGTM. Just a small clean-up left to do.

Note that `pos_label` is not needed for roc_auc because it is the same regardless of `pos_label`, but it is needed for `average_precsion_score` (which will also use the `_check_pos_label_statistics` helper)

The `_check_auc` function is still also defined above the test function but not used anymore. I think you can remove it

```suggestion
    """Test that switching `pos_label` gives correct statistics, using imbalanced data."""
```

```suggestion
    # create highly imbalanced classes
```

Out of curiosity: Why it is harder with only 2 features?
I see that this PR did not change that.

```suggestion
    # Sanity check to be sure the positive class is `classes_[0]`.
    # Class imbalance ensures a large difference in prediction values between classes,
    # allowing us to catch errors when we switch `pos_label`.
    assert classifier.classes_.tolist() == ["cancer", "not cancer"]
```

I know these numbers are not from this PR. Still, I'm curious where they come from.

Amended, but couldn't use all the words suggested to fit allow it to fit one line

Honestly I think only [USER_REDACTED] can (best) answer this question.

Here I think it means that the classification problem is harder if we only include 2 features. And with highly imbalanced classes, a harder classification problem may make the model lean towards predicting the majority class with more confidence (since it does not have enough features to make 'good' predictions)?

But I was confused too: https://github.com/[USER_REDACTED]n/scikit-learn/pull/30399#discussion_r1917786538

This may sounds bad, but I guess this is effectively a non-regression test - the numbers are literally the calculated roc auc values.

I assume the single curve one (L836) was added by [USER_REDACTED] who added this test.

The multi curve ones were added in #30399 by me, and follow the same procedure.

https://github.com/[USER_REDACTED]n/scikit-learn/pull/30399/files#diff-a455a8aae5c9b59034d2e163bbfc31a21de369eb8b53f3c7e5bddd684eeabfd1L329


FIX add missing random states and cloning in `test_response.py`

<!--
🙌 Thanks for contributing a pull request!

👀 Please ensure you have taken a look at the contribution guidelines:
https://github.com/[USER_REDACTED]n/scikit-learn/blob/main/CONTRIBUTING.md

✅ In particular following the pull request checklist will increase the likelihood
of having maintainers review your PR:
https://scikit-learn.org/dev/developers/contributing.html#pull-request-checklist

📋 If your PR is likely to affect users, you will need to add a changelog entry
describing your PR changes, see:
https://github.com/[USER_REDACTED]n/scikit-learn/blob/main/doc/whats_new/upcoming_changes/README.md
-->

#### [NAME_REDACTED]/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Towards #32393 and possibly also #33216

#### What does this implement/fix? Explain your changes.
I added the missing `random_seed` to the `KMeans` test cases and also added `clone` before `fit` (as it is done for example in line 240) where it was missing.

[USER_REDACTED]
#### AI usage disclosure
<!--
If AI tools were involved in creating this PR, please check all boxes that apply
below and make sure that you adhere to our [NAME_REDACTED]:
https://scikit-learn.org/dev/developers/contributing.html#automated-contributions-policy
-->
I used AI assistance for:
- [ ] Code generation (e.g., when writing an implementation or fixing a bug)
- [ ] Test/benchmark generation
- [ ] Documentation (including examples)
- [ ] Research and understanding

#### Any other comments?

<!--
Thank you for your patience. Changes to scikit-learn require careful
attention, but with limited maintainer time, not every contribution can be reviewed
quickly.
For more information and tips on improving your pull request, see:
https://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

@@ -16,7 +16,7 @@
 from sklearn.preprocessing import scale
 from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor
 from sklearn.utils._response import _get_response_values, _get_response_values_binary
-from sklearn.utils._testing import assert_allclose, assert_array_equal
+from sklearn.utils._testing import assert_allclose

 X, y = load_iris(return_X_y=True)
 # scale the data to avoid ConvergenceWarning with LogisticRegression
@@ -29,17 +29,17 @@
     [
         (DecisionTreeRegressor(), "predict_proba"),
         (DecisionTreeRegressor(), ["predict_proba", "decision_function"]),
-        (KMeans(n_clusters=2, n_init=1), "predict_proba"),
-        (KMeans(n_clusters=2, n_init=1), ["predict_proba", "decision_function"]),
+        (KMeans(n_clusters=2), "predict_proba"),
+        (KMeans(n_clusters=2), ["predict_proba", "decision_function"]),
         (DBSCAN(), "predict"),
-        (IsolationForest(random_state=0), "predict_proba"),
-        (IsolationForest(random_state=0), ["predict_proba", "score"]),
+        (IsolationForest(), "predict_proba"),
+        (IsolationForest(), ["predict_proba", "score"]),
     ],
 )
-def test_estimator_unsupported_response(pyplot, estimator, response_method):
+def test_estimator_unsupported_response(estimator, response_method):
     """Check the error message with not supported response method."""
     X, y = np.random.RandomState(0).randn(10, 2), np.array([0, 1] * 5)
-    estimator.fit(X, y)
+    estimator = clone(estimator).fit(X, y)  # clone to make test execution thread-safe
     err_msg = "has none of the following attributes:"
     with pytest.raises(AttributeError, match=err_msg):
         _get_response_values(
@@ -53,9 +53,9 @@ def test_estimator_unsupported_response(pyplot, estimator, response_method):
     "estimator, response_method",
     [
         (LinearRegression(), "predict"),
-        (KMeans(n_clusters=2, n_init=1), "predict"),
-        (KMeans(n_clusters=2, n_init=1), "score"),
-        (KMeans(n_clusters=2, n_init=1), ["predict", "score"]),
+        (KMeans(n_clusters=2, random_state=0), "predict"),
+        (KMeans(n_clusters=2, random_state=0), "score"),
+        (KMeans(n_clusters=2, random_state=0), ["predict", "score"]),
         (IsolationForest(random_state=0), "predict"),
         (IsolationForest(random_state=0), "decision_function"),
         (IsolationForest(random_state=0), ["decision_function", "predict"]),
@@ -67,7 +67,7 @@ def test_estimator_get_response_values(
 ):
     """Check the behaviour of `_get_response_values`."""
     X, y = np.random.RandomState(0).randn(10, 2), np.array([0, 1] * 5)
-    estimator.fit(X, y)
+    estimator = clone(estimator).fit(X, y)  # clone to make test execution thread-safe
     results = _get_response_values(
         estimator,
         X,
@@ -78,7 +78,7 @@ def test_estimator_get_response_values(
         response_method[0] if isinstance(response_method, list) else response_method
     )
     prediction_method = getattr(estimator, chosen_response_method)
-    assert_array_equal(results[0], prediction_method(X))
+    assert_allclose(results[0], prediction_method(X))
     assert results[1] is None
     if return_response_method_used:
         assert results[2] == chosen_response_method
@@ -310,8 +310,7 @@ def test_get_response_values_multiclass(estimator, response_method):
     """Check that we can call `_get_response_values` with a multiclass estimator.
     It should return the predictions untouched.
     """
-    estimator = clone(estimator)
-    estimator.fit(X, y)
+    estimator = clone(estimator).fit(X, y)  # clone to make test execution thread-safe
     predictions, pos_label = _get_response_values(
         estimator, X, response_method=response_method
     )
@@ -416,8 +415,8 @@ def test_response_values_type_of_target_on_classes_no_warning():
         (IsolationForest(), "predict", "multiclass", (10,)),
         (DecisionTreeRegressor(), "predict", "binary", (10,)),
         (DecisionTreeRegressor(), "predict", "multiclass", (10,)),
-        (KMeans(n_clusters=2, n_init=1), "predict", "binary", (10,)),
-        (KMeans(n_clusters=2, n_init=1), "predict", "multiclass", (10,)),
+        (KMeans(n_clusters=2), "predict", "binary", (10,)),
+        (KMeans(n_clusters=4), "predict", "multiclass", (10,)),
     ],
 )
 def test_response_values_output_shape_(
@@ -442,8 +441,8 @@ def test_response_values_output_shape_(
     else:  # multilabel
         y = np.array([[0, 1], [1, 0]] * 5)

-    clf = clone(estimator).fit(X, y)
+    estimator = clone(estimator).fit(X, y)  # clone to make test execution thread-safe

-    y_pred, _ = _get_response_values(clf, X, response_method=response_method)
+    y_pred, _ = _get_response_values(estimator, X, response_method=response_method)

     assert y_pred.shape == expected_shape

LGTM, thanks!

This fixes all the failures on free-threaded locally for me and cleans up a few things.

LGTM. Thanks !

Unrelated small clean-up: changed `clf` to`estimator` because different estimators are tested here

And I added the comment on explaining the cloning here as well, because I didn't know this, and it probably doesn't hurt to be expressive on this.

Unrelated small clean-up: remove redundant parameter and make test case more consistent

Here we just check the attributes, so setting the `random_state` has no impact. Not sure if we should consistently add or remove it in such cases?

`n_init=1` is the default value in this case anyways

I think it's fine not to have `random_state` in this case.

Note: Here we also just check response shapes, so random states don't matter for this test.

Just curious, you are doing this for free-threaded because you have noticed issues when running the test locally with pytest-run-parallel?

No, I saw that this was done like this with this comment on line 240 (added by Olivier) and adapted it for the rest.

is changing to `n_clusters=4` on purpose ?

Yes, to make explicit that the output shape also doesn't change in the multiclass (or rather, "multicluster") case. I didn't see this in the PR where it was added, but I can also make a separate PR for the cleanu-ps if this is becoming too cluttered now.

Thanks for the clarification. It's fine to keep it in this PR