DOC Fix response values shape

<!--
🙌 Thanks for contributing a pull request!

👀 Please ensure you have taken a look at the contribution guidelines:
https://github.com/[USER_REDACTED]n/scikit-learn/blob/main/CONTRIBUTING.md

✅ In particular following the pull request checklist will increase the likelihood
of having maintainers review your PR:
https://scikit-learn.org/dev/developers/contributing.html#pull-request-checklist

📋 If your PR is likely to affect users, you will need to add a changelog entry
describing your PR changes, see:
https://github.com/[USER_REDACTED]n/scikit-learn/blob/main/doc/whats_new/upcoming_changes/README.md
-->

#### [NAME_REDACTED]/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Towards #32872

#### What does this implement/fix? Explain your changes.
The problem in #32872 arises from the assumption that multiclass classification will have a different output shape than binary classification, which is not the case when `response_method`="predict".
To make sure, I added a test for all output shapes and updated the docstring according to the result.

#### AI usage disclosure
<!--
If AI tools were involved in creating this PR, please check all boxes that apply
below and make sure that you adhere to our [NAME_REDACTED]:
https://scikit-learn.org/dev/developers/contributing.html#automated-contributions-policy
-->
I used AI assistance for:
- [ ] Code generation (e.g., when writing an implementation or fixing a bug)
- [ ] Test/benchmark generation
- [ ] Documentation (including examples)
- [ ] Research and understanding

#### Any other comments?

I have not checked if this assumption is used anywhere else.

<!--
Thank you for your patience. Changes to scikit-learn require careful
attention, but with limited maintainer time, not every contribution can be reviewed
quickly.
For more information and tips on improving your pull request, see:
https://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

@@ -125,7 +125,9 @@ def _get_response_values(
     The response values are predictions such that it follows the following shape:

     - for binary classification, it is a 1d array of shape `(n_samples,)`;
-    - for multiclass classification, it is a 2d array of shape `(n_samples, n_classes)`;
+    - for multiclass classification
+        - with response_method="predict", it is a 1d array of shape `(n_samples,)`;
+        - otherwise, it is a 2d array of shape `(n_samples, n_classes)`;
     - for multilabel classification, it is a 2d array of shape `(n_samples, n_outputs)`;
     - for outlier detection, it is a 1d array of shape `(n_samples,)`;
     - for regression, it is a 1d array of shape `(n_samples,)`.

@@ -394,3 +394,55 @@ def test_response_values_type_of_target_on_classes_no_warning():
         warnings.simplefilter("error", UserWarning)

         _get_response_values(clf, X, response_method="predict_proba")
+
+
+[USER_REDACTED].mark.parametrize(
+    "estimator, response_method, target_type, expected_shape",
+    [
+        (LogisticRegression(), "predict", "binary", (10,)),
+        (LogisticRegression(), "predict_proba", "binary", (10,)),
+        (LogisticRegression(), "decision_function", "binary", (10,)),
+        (LogisticRegression(), "predict", "multiclass", (10,)),
+        (LogisticRegression(), "predict_proba", "multiclass", (10, 4)),
+        (LogisticRegression(), "decision_function", "multiclass", (10, 4)),
+        (ClassifierChain(LogisticRegression()), "predict", "multilabel", (10, 2)),
+        (ClassifierChain(LogisticRegression()), "predict_proba", "multilabel", (10, 2)),
+        (
+            ClassifierChain(LogisticRegression()),
+            "decision_function",
+            "multilabel",
+            (10, 2),
+        ),
+        (IsolationForest(), "predict", "binary", (10,)),
+        (IsolationForest(), "predict", "multiclass", (10,)),
+        (DecisionTreeRegressor(), "predict", "binary", (10,)),
+        (DecisionTreeRegressor(), "predict", "multiclass", (10,)),
+    ],
+)
+def test_response_values_output_shape_(
+    estimator, response_method, target_type, expected_shape
+):
+    """
+    Check that output shape corresponds to docstring description
+
+    - for binary classification, it is a 1d array of shape `(n_samples,)`;
+    - for multiclass classification
+        - with response_method="predict", it is a 1d array of shape `(n_samples,)`;
+        - otherwise, it is a 2d array of shape `(n_samples, n_classes)`;
+    - for multilabel classification, it is a 2d array of shape `(n_samples, n_outputs)`;
+    - for outlier detection, it is a 1d array of shape `(n_samples,)`;
+    - for regression, it is a 1d array of shape `(n_samples,)`.
+    """
+    X = np.random.RandomState(0).randn(10, 2)
+    if target_type == "binary":
+        y = np.array([0, 1] * 5)
+    elif target_type == "multiclass":
+        y = [0, 1, 2, 3, 0, 1, 2, 3, 3, 0]
+    else:  # multilabel
+        y = np.array([[0, 1], [1, 0]] * 5)
+
+    clf = estimator.fit(X, y)
+
+    y_pred, _ = _get_response_values(clf, X, response_method=response_method)
+
+    assert y_pred.shape == expected_shape

LGTM

LGTM! Thanks, [USER_REDACTED]!

[USER_REDACTED], could we merge this?

This is the test that was inconsistent with the previous version of the docstring (which suggested that it should be `(10,4)`)

Can the test values for `y` be something like `[0, 1, 2, 3, 0, 1, 2, 3, 3, 0]` so that the test would check for `(10, 4)` for the `multi-class` + `predict_proba` case?

This makes it more obvious that `n_samples` and `n_classes` can be different values.


FIX Error handling in ranking metrics supporting multiclass: `average_precision_score`, `roc_auc_score` and `top_k_accuracy_score`

#### [NAME_REDACTED]/PRs
discovered while working in #32909

#### What does this implement/fix? Explain your changes.
This adds a proper error handling if a multiclass `y_true` is passed into `average_precision_score`, `roc_auc_score` and `top_k_accuracy_score` while `y_score` has only shape `(n_classes, )` (aka lacks information on the non-dominant classes).

`y_true`  is internally label-binarized to make it match a multi-label case.

There is no way to calculate `y_score` from a binarised `y_true`, since we lack information on the scores of the other classes.
Thus, `y_score` needs to be passed with shape `(n_samples, n_classes)`.

The previous error raised in `average_precision_score` from `check_array` for instance was
```
E ValueError: Expected 2D array, got 1D array instead:
E array=[0.5 0.4 0.8 0.9 0.8 0.7].
E Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.
```
which gives wrong advise. Users cannot deal with that by reshaping `y_score`.

#### AI usage disclosure
<!--
If AI tools were involved in creating this PR, please check all boxes that apply
below and make sure that you adhere to our [NAME_REDACTED]:
https://scikit-learn.org/dev/developers/contributing.html#automated-contributions-policy
-->
I used AI assistance for:
- [ ] Code generation (e.g., when writing an implementation or fixing a bug)
- [ ] Test/benchmark generation
- [ ] Documentation (including examples)
- [x] Research and understanding

#### Any other comments?
Discovered while adding the `check_array_api_multiclass_classification_metric` test case in #32909.

Also added some documentation on the multiclass case for `average_precision_score`, since it was lacking.

@@ -142,7 +142,8 @@ def average_precision_score(
     Parameters
     ----------
     y_true : array-like of shape (n_samples,) or (n_samples, n_classes)
-        True binary labels or :term:`multilabel indicator matrix`.
+        True binary labels, :term:`multi-label` indicators (as a
+        :term:`multilabel indicator matrix`) or :term:`multi-class` labels.

     y_score : array-like of shape (n_samples,) or (n_samples, n_classes)
         Target scores, can either be probability estimates of the positive
@@ -261,6 +262,12 @@ def _binary_uninterpolated_average_precision(
                 "Do not set pos_label or set pos_label to 1."
             )
         y_true = label_binarize(y_true, classes=present_labels)
+        if not y_score.shape == y_true.shape:
+            raise ValueError(
+                "`y_score` needs to be of shape `(n_samples, n_classes)`, since "
+                "`y_true` contains multiple classes. Got "
+                f"`y_score.shape={y_score.shape}`."
+            )

     average_precision = partial(
         _binary_uninterpolated_average_precision, pos_label=pos_label
@@ -764,7 +771,12 @@ def _multiclass_roc_auc_score(
         Sample weights.

     """
-    # validation of the input y_score
+    if not y_score.ndim == 2:
+        raise ValueError(
+            "`y_score` needs to be of shape `(n_samples, n_classes)`, since "
+            "`y_true` contains multiple classes. Got "
+            f"`y_score.shape={y_score.shape}`."
+        )
     if not np.allclose(1, y_score.sum(axis=1)):
         raise ValueError(
             "Target scores need to be probabilities for multiclass "
@@ -2111,6 +2123,13 @@ def top_k_accuracy_score(
                 " labels, `labels` must be provided."
             )
         y_score = column_or_1d(y_score)
+    else:
+        if not y_score.ndim == 2:
+            raise ValueError(
+                "`y_score` needs to be of shape `(n_samples, n_classes)`, since "
+                "`y_true` contains multiple classes. Got "
+                f"`y_score.shape={y_score.shape}`."
+            )

     check_consistent_length(y_true, y_score, sample_weight)
     y_score_n_classes = y_score.shape[1] if y_score.ndim == 2 else 2

@@ -1212,7 +1212,7 @@ def test_average_precision_score_multilabel_pos_label_errors():
 def test_average_precision_score_multiclass_pos_label_errors():
     # Raise an error for multiclass y_true with pos_label other than 1
     y_true = np.array([0, 1, 2, 0, 1, 2])
-    y_pred = np.array(
+    y_score = np.array(
         [
             [0.5, 0.2, 0.1],
             [0.4, 0.5, 0.3],
@@ -1227,7 +1227,21 @@ def test_average_precision_score_multiclass_pos_label_errors():
         "Do not set pos_label or set pos_label to 1."
     )
     with pytest.raises(ValueError, match=err_msg):
-        average_precision_score(y_true, y_pred, pos_label=3)
+        average_precision_score(y_true, y_score, pos_label=3)
+
+
+def test_multiclass_ranking_metrics_raise_for_incorrect_shape_of_y_score():
+    """Test ranking metrics, with multiclass support, raise if shape `y_score` is 1D."""
+    y_true = np.array([0, 1, 2, 0, 1, 2])
+    y_score = np.array([0.5, 0.4, 0.8, 0.9, 0.8, 0.7])
+
+    msg = re.escape("`y_score` needs to be of shape `(n_samples, n_classes)`")
+    with pytest.raises(ValueError, match=msg):
+        average_precision_score(y_true, y_score)
+    with pytest.raises(ValueError, match=msg):
+        roc_auc_score(y_true, y_score, multi_class="ovr")
+    with pytest.raises(ValueError, match=msg):
+        top_k_accuracy_score(y_true, y_score)

 def test_score_scale_invariance():

I think some other metrics in `_ranking.py` may also benefit from this extra check, even if just for consistency -  e.g., `roc_auc_score` gives the following error (with the input from your new test), which is less informative:

```
AxisError: axis 1 is out of bounds for array of dimension 1
```

Otherwise looks good.

Do you think it is worth factorising this check out? Mostly so we can easily ensure the error message is always consistent, if it ever gets changed...

LGTM!

LGTM! Thanks, [USER_REDACTED]!

I only have a few optional suggestions, but feel free to merge this as is.

Renamed it to match the param name.

We actually have a glossary term for indicator format: see #32893

I agree that it would be nice to specify that it is multilabel type should be in indicator format, but it would also be nice to have a glossary reference to both multilabel and label indicator here - I think can manage both

I didn't see your PR before. Will have a look. Do you want to include this suggestion in it and try to fit both links to multilabel and label indicator in there? I also think it is useful to link to both.

```suggestion
    """Test ranking metrics, with multiclass support, raises if shape `y_score` is 1D"""
```
nit, to make it fit on one line, as per docstring standard

Hmm maybe it's better in this PR as it fits with the content here more? It does mean some minor merge conflict resolving but it would only be one line.

I don't mind to have it here. Will also add the other reference and from your PR.

```suggestion
                f"`{y_score.shape = }`."
```


FEA Add array API support for `average_precision_score`

#### [NAME_REDACTED]/PRs
towards #26024

#### What does this implement/fix? Explain your changes.
Adds array API support to `average_precision_score`

#### AI usage disclosure
<!--
If AI tools were involved in creating this PR, please check all boxes that apply
below and make sure that you adhere to our [NAME_REDACTED]:
https://scikit-learn.org/dev/developers/contributing.html#automated-contributions-policy
-->
I used AI assistance for:
- [ ] Code generation (e.g., when writing an implementation or fixing a bug)
- [ ] Test/benchmark generation
- [ ] Documentation (including examples)
- [x] Research and understanding

#### Any other comments?
WIP

TODO:
- [x] add array API in `average_precision_score`
- [x] add common metrics test
- [x] add average_precision_score to the array API docs
- [x] removed old fix for numpy<= 1.24.1 from `average_precision_score` as well as from `_check_set_wise_labels` which allows to have `present_labels` as an array instead of a list
- [x] use `LabelBinarizer` instead of `label_binarize` # TODO: check if in fact necessary
- [x] add array API in `_average_binary_score`
- [x] fix remaining test failures
- [x] add changelog
- [x] make it accept mixed namespace inputs
- [ ] (only necessary if #32755 is merged first) add new test case for `average_precision_score` in `METRICS_SUPPORTING_MIXED_NAMESPACE` in `sklearn/metrics/tests/test_common.py`

@@ -160,6 +160,7 @@ Metrics
 -------

 - :func:`sklearn.metrics.accuracy_score`
+- :func:`sklearn.metrics.average_precision_score`
 - :func:`sklearn.metrics.balanced_accuracy_score`
 - :func:`sklearn.metrics.brier_score_loss`
 - :func:`sklearn.metrics.cluster.calinski_harabasz_score`

@@ -0,0 +1,3 @@
+- :func:`sklearn.metrics.ranking.average_precision_score` now supports Array API
+  compliant inputs.
+  By :user:`[NAME_REDACTED] <StefanieSenger>`.

@@ -10,7 +10,9 @@

 import numpy as np

+import sklearn.externals.array_api_extra as xpx
 from sklearn.utils import check_array, check_consistent_length
+from sklearn.utils._array_api import _average, _ravel, get_namespace_and_device
 from sklearn.utils.multiclass import type_of_target

@@ -19,6 +21,9 @@ def _average_binary_score(binary_metric, y_true, y_score, average, sample_weight

     Parameters
     ----------
+    binary_metric : callable, returns shape [n_classes]
+        The binary metric function to use.
+
     y_true : array, shape = [n_samples] or [n_samples, n_classes]
         True binary labels in binary label indicators.

@@ -47,16 +52,14 @@ def _average_binary_score(binary_metric, y_true, y_score, average, sample_weight
     sample_weight : array-like of shape (n_samples,), default=None
         Sample weights.

-    binary_metric : callable, returns shape [n_classes]
-        The binary metric function to use.
-
     Returns
     -------
     score : float or array of shape [n_classes]
         If not ``None``, average the score, else return the score for each
         classes.

     """
+    xp, _, _device = get_namespace_and_device(y_true, y_score, sample_weight)
     average_options = (None, "micro", "macro", "weighted", "samples")
     if average not in average_options:
         raise ValueError("average has to be one of {0}".format(average_options))
@@ -78,18 +81,23 @@ def _average_binary_score(binary_metric, y_true, y_score, average, sample_weight

     if average == "micro":
         if score_weight is not None:
-            score_weight = np.repeat(score_weight, y_true.shape[1])
-        y_true = y_true.ravel()
-        y_score = y_score.ravel()
+            score_weight = xp.repeat(score_weight, y_true.shape[1])
+        y_true = _ravel(y_true)
+        y_score = _ravel(y_score)

     elif average == "weighted":
         if score_weight is not None:
-            average_weight = np.sum(
-                np.multiply(y_true, np.reshape(score_weight, (-1, 1))), axis=0
+            #  Mixed integer and float type promotion not defined in array standard
+            y_true = xp.asarray(y_true, dtype=score_weight.dtype)
+            average_weight = xp.sum(
+                xp.multiply(y_true, xp.reshape(score_weight, (-1, 1))), axis=0
             )
         else:
-            average_weight = np.sum(y_true, axis=0)
-        if np.isclose(average_weight.sum(), 0.0):
+            average_weight = xp.sum(y_true, axis=0)
+        if xpx.isclose(
+            xp.sum(average_weight),
+            xp.asarray(0, dtype=average_weight.dtype, device=_device),
+        ):
             return 0

     elif average == "samples":
@@ -99,26 +107,29 @@ def _average_binary_score(binary_metric, y_true, y_score, average, sample_weight
         not_average_axis = 0

     if y_true.ndim == 1:
-        y_true = y_true.reshape((-1, 1))
+        y_true = xp.reshape(y_true, (-1, 1))

     if y_score.ndim == 1:
-        y_score = y_score.reshape((-1, 1))
+        y_score = xp.reshape(y_score, (-1, 1))

     n_classes = y_score.shape[not_average_axis]
-    score = np.zeros((n_classes,))
+    score = xp.zeros((n_classes,), device=_device)
     for c in range(n_classes):
-        y_true_c = y_true.take([c], axis=not_average_axis).ravel()
-        y_score_c = y_score.take([c], axis=not_average_axis).ravel()
+        y_true_c = _ravel(
+            xp.take(y_true, xp.asarray([c], device=_device), axis=not_average_axis)
+        )
+        y_score_c = _ravel(
+            xp.take(y_score, xp.asarray([c], device=_device), axis=not_average_axis)
+        )
         score[c] = binary_metric(y_true_c, y_score_c, sample_weight=score_weight)

     # Average the results
     if average is not None:
         if average_weight is not None:
             # Scores with 0 weights are forced to be 0, preventing the average
             # score from being affected by 0-weighted NaN elements.
-            average_weight = np.asarray(average_weight)
             score[average_weight == 0] = 0
-        return float(np.average(score, weights=average_weight))
+        return float(_average(score, weights=average_weight, xp=xp))
     else:
         return score

@@ -31,6 +31,7 @@
 from sklearn.utils._array_api import (
     _max_precision_float_dtype,
     get_namespace_and_device,
+    move_to,
     size,
 )
 from sklearn.utils._encode import _encode, _unique
@@ -225,25 +226,36 @@ def average_precision_score(
     >>> average_precision_score(y_true, y_scores)
     0.77
     """
+    xp, _, device = get_namespace_and_device(y_score)
+    y_true, sample_weight = move_to(y_true, sample_weight, xp=xp, device=device)
+
+    if sample_weight is not None:
+        sample_weight = column_or_1d(sample_weight)

     def _binary_uninterpolated_average_precision(
-        y_true, y_score, pos_label=1, sample_weight=None
+        y_true,
+        y_score,
+        pos_label=1,
+        sample_weight=None,
+        xp=xp,
     ):
         precision, recall, _ = precision_recall_curve(
-            y_true, y_score, pos_label=pos_label, sample_weight=sample_weight
+            y_true,
+            y_score,
+            pos_label=pos_label,
+            sample_weight=sample_weight,
         )
         # Return the step function integral
         # The following works because the last entry of precision is
         # guaranteed to be 1, as returned by precision_recall_curve.
         # Due to numerical error, we can get `-0.0` and we therefore clip it.
-        return float(max(0.0, -np.sum(np.diff(recall) * np.array(precision)[:-1])))
+        return float(max(0.0, -xp.sum(xp.diff(recall) * precision[:-1])))

     y_type = type_of_target(y_true, input_name="y_true")
-
-    present_labels = np.unique(y_true)
+    present_labels = xp.unique_values(y_true)

     if y_type == "binary":
-        if len(present_labels) == 2 and pos_label not in present_labels:
+        if present_labels.shape[0] == 2 and pos_label not in present_labels:
             raise ValueError(
                 f"pos_label={pos_label} is not a valid label. It should be "
                 f"one of {present_labels}"
@@ -270,7 +282,7 @@ def _binary_uninterpolated_average_precision(
             )

     average_precision = partial(
-        _binary_uninterpolated_average_precision, pos_label=pos_label
+        _binary_uninterpolated_average_precision, pos_label=pos_label, xp=xp
     )
     return _average_binary_score(
         average_precision, y_true, y_score, average, sample_weight=sample_weight
@@ -686,6 +698,8 @@ class scores must correspond to the order of ``labels``,
     y_type = type_of_target(y_true, input_name="y_true")
     y_true = check_array(y_true, ensure_2d=False, dtype=None)
     y_score = check_array(y_score, ensure_2d=False)
+    if sample_weight is not None:
+        sample_weight = column_or_1d(sample_weight)

     if y_type == "multiclass" or (
         y_type == "binary" and y_score.ndim == 2 and y_score.shape[1] > 2
@@ -1142,7 +1156,7 @@ def precision_recall_curve(
             "No positive class found in y_true, "
             "recall is set to one for all thresholds."
         )
-        recall = xp.full(tps.shape, 1.0)
+        recall = xp.full(tps.shape, 1.0, device=device)
     else:
         recall = tps / tps[-1]

@@ -2098,6 +2098,18 @@ def check_array_api_multiclass_classification_metric(
     y_true_np = np.array([0, 1, 2, 3])
     y_pred_np = np.array([0, 1, 0, 2])

+    if metric.__name__ == "average_precision_score":
+        # we need y_pred_nd to be of shape (n_samples, n_classes)
+        y_pred_np = np.array(
+            [
+                [0.7, 0.2, 0.05, 0.05],
+                [0.1, 0.8, 0.05, 0.05],
+                [0.1, 0.1, 0.7, 0.1],
+                [0.05, 0.05, 0.1, 0.8],
+            ],
+            dtype=dtype_name,
+        )
+
     additional_params = {
         "average": ("micro", "macro", "weighted"),
         "beta": (0.2, 0.5, 0.8),
@@ -2299,6 +2311,11 @@ def check_array_api_metric_pairwise(metric, array_namespace, device, dtype_name)
         check_array_api_multiclass_classification_metric,
         check_array_api_multilabel_classification_metric,
     ],
+    average_precision_score: [
+        check_array_api_binary_classification_metric,
+        check_array_api_multiclass_classification_metric,
+        check_array_api_multilabel_classification_metric,
+    ],
     balanced_accuracy_score: [
         check_array_api_binary_classification_metric,
         check_array_api_multiclass_classification_metric,

Just a question and a nit.

Thank you [USER_REDACTED], [USER_REDACTED] and [USER_REDACTED]. I have now used `column_or_1d(sample_weight)` for an early `check_array` on it (see https://github.com/[USER_REDACTED]n/scikit-learn/pull/32909#discussion_r2701343355) and also removed the special casing of array_api_strict in `xp.multiply` (see https://github.com/[USER_REDACTED]n/scikit-learn/pull/32909#discussion_r2701298993).

There is one open question whether we should do

```py
        y_true_c = _ravel(
            xp.take(y_true, xp.asarray([c], device=_device), axis=not_average_axis)
         )
        y_score_c = _ravel(
            xp.take(y_score, xp.asarray([c], device=_device), axis=not_average_axis)
        )
```

or

```py
        y_true_rotated = xp.moveaxis(y_true, not_average_axis, -1)
        y_true_c = _ravel(y_true_rotated[..., c])
        y_score_rotated = xp.moveaxis(y_score, not_average_axis, -1)
        y_score_c = _ravel(y_score_rotated[..., c])
```

or something else (see https://github.com/[USER_REDACTED]n/scikit-learn/pull/32909#discussion_r2701341753).

As far as I can tell, LGTM.

LGTM as well! Thanks, [USER_REDACTED]!

Just one minor nitpick.

Also, perhaps [USER_REDACTED] or [USER_REDACTED] could take a look before merging.

Nits and a question, otherwise looks good!

This change is a consequence of `present_labels` now being an array, not a list.
See [this comment](https://github.com/[USER_REDACTED]n/scikit-learn/pull/32909/changes#r2623035023).

I changed the order of the params in the docs to match the order of reading these into the function, like in all the other places.

This had been fixed three years ago, see https://github.com/numpy/numpy/issues/6784#issuecomment-1341719384. It makes sense to keep `present_labels` as an array instead of turning it into a list.

I suppose this fix went into numpy 1.24.1 or even earlier.

This case handling needs to be added, because `average_precision_score` handels multiclass inputs by [NAME_REDACTED] a consequence, y_score needs to be of shape `(n_samples, n_classes)`. I have made a PR with a fix here: #32912.

This prevents an error from `array_api_strict`:

`TypeError: array_api_strict.int64 and array_api_strict.float64 cannot be type promoted together`

But actually, we should not change our code just to make a helper library happy. I need to think a bit more about this.

This is not needed anymore, since `average_weight` is converted to an array earlier.

This (hopefully) fixes an error occurring on the CI ([link to output)](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=83504&view=logs&j=dde5042c-7464-5d47-9507-31bdd2ee0a3a&t=4bd2dad8-62b3-5bf9-08a5-a9880c530c94):

```
E TypeError: sum() received an invalid combination of arguments - got (out=NoneType, axis=int, ), but expected one of:
E * (*, torch.dtype dtype = None)
E  didn't match because some of the keywords were incorrect: out, axis
E  * (tuple of ints dim, bool keepdim = False, *, torch.dtype dtype = None)
E  * (tuple of names dim, bool keepdim = False, *, torch.dtype dtype = None)
```

[USER_REDACTED] and I investigated this together. It seems that `numpy.sum() `when passed a torch tensor, internally tries to dispatch to torch, but that fails because `torch.sum() `doesn't have the `out` argument. The same happens with `np.repeat()` which has an `axis` argument, but `torch.repeat()` hasn't (which had raised in my setup locally).

`_average_binary_score` is also used in `roc_auc_score`. So it's probably best to first discuss how to properly add array API support to it, or wait until this PR is merged before creating a PR for `roc_auc_score`.

```suggestion
from sklearn.utils._array_api import _ravel, get_namespace_and_device, xpx
```

Yes, I ran into the same issue when I was working on `roc_auc_score`.

I made a separate PR for it and it is merged: #32917.
This comment is resolved here.

I wasn't aware of that, and maybe that had been discussed somewhere I missed. So I did a bit of searching and I have found `xpx` imported via `_array_api.py` in `sklearn/model_selection/_search.py` and in `sklearn/tree/tests/test_tree.py`.
And I have found `from sklearn.externals import array_api_extra as xpx` imports in three other files.

I prefer the more explicit import from `sklearn.externals`, since it is easier to see what is going on the first glance even if you are unfamiliar with array api, and it will have to be obvious for people unfamilar with array api, as it will be in every function in the future and we also want to make it easy for future contributors.

Instead of converting everything to NumPy ahead of time, we should inspect the namespace and device of `y_score` only. Indeed, we want to allow `y_true` to have an object dtype for string class labels (hence a NumPy array) even when the predictions are a non-NumPy namespace with a non-CPU device.

See: https://scikit-learn.org/dev/modules/array_api.html#scoring-functions

`y_true` should thereafter convert to binary indicators (using `pos_label` when available, and moved to the same device, dtype and namespace as `y_score` just before doing the arithmetic computation.

`sample_weight` should similarly follow the namespace, dtype and device of `y_score` using `move_to`.

See the discussion and implementation of the common test being worked on in #32755 for mixed input metrics.

I will keep your comment in mind for later, but this here addresses a different problem. It fixes a test failure if array api is **not** enabled and all inputs (specifically `y_true` and `y_score`) are torch tensors on cpu.

In `check_array_api_metric()` we have defined a `numpy_as_array_works` boolean that is `True` if array api is not enabled, but we get input from a non-numpy namespace that can be converted to numpy (see [here](https://github.com/[USER_REDACTED]n/scikit-learn/blob/4ac107924f6c017481c1a8c432c0512a9ec2dc0a/sklearn/metrics/tests/test_common.py#L1999-L2006)):

```py
    # When array API dispatch is disabled, and np.asarray works (for example PyTorch
    # with CPU device), calling the metric function with such numpy compatible inputs
    # should work (albeit by [NAME_REDACTED] arrays instead of
    # dispatching to the array library).
    try:
        np.asarray(a_xp)
        np.asarray(b_xp)
        numpy_as_array_works = True
```

In this case, we feed two torch arrays into the metric, but the test fails (see [here](https://github.com/[USER_REDACTED]n/scikit-learn/blob/4ac107924f6c017481c1a8c432c0512a9ec2dc0a/sklearn/metrics/tests/test_common.py#L2023-L2024)):

```py
    if numpy_as_array_works:
        metric_xp = metric(a_xp, b_xp, **metric_kwargs)
```

If I have understood correctly, then we would always try to convert to numpy, `if get_config().get("array_api_dispatch", False)` but the input would be convertible to numpy, and making sure that we do is the purpose of this test (added in #30454).

(In this WIP PR I haven't taken care of mixed inputs at all yet, also #32755 is not merged yet.
See:
<img width="1108" height="438" alt="image" src="https://github.com/[USER_REDACTED]nts/assets/f256acd5-d3df-40ac-af9d-647d76e848dc" />
)

Sorry, I didn't follow what has been fixed. Did we test for a Future warning?

Maybe we could add a comment about why we are doing this?

Just to clarify, what was previously doing the asarray for torch inputs? `type_of_target` does do `check_array`, but does that not fix the error?

This comment is resolved. I've fixed this in a separate PR (#32917).

> Just to clarify, what was previously doing the asarray for torch inputs?

The asarray works as expected, no problems here.

The error raised in:

https://github.com/[USER_REDACTED]n/scikit-learn/blob/938cd9798879757d8f681d5c82cdf0503aac96a3/sklearn/metrics/_base.py#L87-L89

when a torch tensor was passed (because then `np.sum()` internally calls torch.sum, which doesn't take an `out` argument.

I've tried to explain this above:

>  It seems that numpy.sum() when passed a torch tensor, internally tries to dispatch to torch, but that fails because torch.sum() doesn't have the out argument.

It is a known bug in numpy. [USER_REDACTED] found this related issue: [numpy/numpy/#28096.](https://github.com/numpy/numpy/issues/28024).
Same for `np.repeat` (when passed a torch tensor), which raised locally for me.

I will add a comment pointing to it.

I've added a comment on this in https://github.com/[USER_REDACTED]n/scikit-learn/pull/32909/changes/9d8bd4632a55e17cdfc28a43637f0d42e628ff7a.

Sorry my confusion is more that I thought previously (and now with dispatch off) we converted all inputs to numpy (also as we generally allow lists, the `np.array` or `np.asarray` call was necessary)?
Which input is a torch tensor...?

Digging into it, it seems that `precision_recall_curve` -> `confusion_matrix_at_thresholds` which calls `column_or_1d` on `y_true` and `y_score`, which does a `check_array`, which I thought ends up calling `_asarray_with_order` and would convert to numpy array (with dispatch disabled)?

<details open>
<summary>Aside</summary>

Aside, I noticed that although we specify inputs can be 'array-like' when the input is indicator matrix, list does not work (checked on main):

```python
from sklearn.metrics import average_precision_score
y_true = [0, 0, 1, 1, 2, 2]
y_scores = [
    [0.7, 0.2, 0.1],
    [0.4, 0.3, 0.3],
    [0.1, 0.8, 0.1],
    [0.2, 0.3, 0.5],
    [0.4, 0.4, 0.2],
    [0.1, 0.2, 0.7],
]
average_precision_score(y_true, y_scores)
```

gives:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[NAME_REDACTED][3], line 1
----> 1 average_precision_score(y_true, y_scores)

File ~/Documents/dev/scikit-learn/sklearn/utils/_param_validation.py:218, in validate_params.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    212 try:
    213     with config_context(
    214         skip_parameter_validation=(
    215             prefer_skip_nested_validation or global_skip_validation
    216         )
    217     ):
--> 218         return func(*args, **kwargs)
    219 except InvalidParameterError as e:
    220     # When the function is just a wrapper around an estimator, we allow
    221     # the function to delegate validation to the estimator, but we replace
    222     # the name of the estimator by [NAME_REDACTED] function in the error
    223     # message to avoid confusion.
    224     msg = re.sub(
    225         r"parameter of \w+ must be",
    226         f"parameter of {func.__qualname__} must be",
    227         str(e),
    228     )

File ~/Documents/dev/scikit-learn/sklearn/metrics/_ranking.py:265, in average_precision_score(y_true, y_score, average, pos_label, sample_weight)
    260         raise ValueError(
    261             "Parameter pos_label is fixed to 1 for multiclass y_true. "
    262             "Do not set pos_label or set pos_label to 1."
    263         )
    264     y_true = label_binarize(y_true, classes=present_labels)
--> 265     if not y_score.shape == y_true.shape:
    266         raise ValueError(
    267             "`y_score` needs to be of shape `(n_samples, n_classes)`, since "
    268             "`y_true` contains multiple classes. Got "
    269             f"`y_score.shape={y_score.shape}`."
    270         )
    272 average_precision = partial(
    273     _binary_uninterpolated_average_precision, pos_label=pos_label
    274 )

AttributeError: 'list' object has no attribute 'shape'
```

This is because in `confusion_matrix_at_thresholds` we do `check_consistent_length` before converting to array via `column_or_1d`.

Edit: I am happy to fix, just wanted to document while I was here.
</details>

> Digging into it, it seems that precision_recall_curve -> confusion_matrix_at_thresholds which calls column_or_1d on y_true and y_score, which does a check_array, which I thought ends up calling _asarray_with_order and would convert to numpy array (with dispatch disabled)?

`precision_recall_curve` indeed does that, but `average_precision_score` does not.

> Sorry my confusion is more that I thought previously (and now with dispatch off) we converted all inputs to numpy (also as we generally allow lists, the np.array or np.asarray call was necessary)?
> Which input is a torch tensor...?

In this test, we check if the inputs (both `y_true` and `y_score`) can be converted to numpy and if they can, we call the metric with `array_api_dispatch=False ` on all the input types:

https://github.com/[USER_REDACTED]n/scikit-learn/blob/3ca712bec1a90f54adc7d7883f0f5f0c6b325e2b/sklearn/metrics/tests/test_common.py#L1999-L2024

This raises on line 2024, where both `a_xp`, `b_xp`, and `sample_weight` in  `metric_kwargs` (if provided) are torch tensors.

A pity the traceback from the CI is now gone, I cannot re-create exactly the same locally (only a similar error concerning `np.repeat`, but let's not complicate this here with the `np.repeat` case, let's first find a common understanding of the problem.)

What happens if you comment out the part of the code we are discussing and run the tests?

Then, the next question is whether this is a good way to fix this, and what are the alternatives.

What I like about this is that it is very explicit. `check_array` would be almost as explicit, but a conversion to numpy happening by [NAME_REDACTED] (somewhere in the code) would be very implicit and dependent on the current implementation. Since we see that we get hard to debug errors, I would not be in favour if this solution.

> precision_recall_curve indeed does that, but average_precision_score does not.

But doesn't `average_precision_score` call `precision_recall_curve` ?

> It fixes a test failure if array api is not enabled and all inputs (specifically y_true and y_score) are torch tensors on cpu.

My understanding of this is that the following code is meant to an error (when dispatch is disabled - explicitly set here):

```python
    import torch
    from sklearn.metrics import average_precision_score

    y_scores = torch.tensor([0.1, 0.4, 0.35, 0.8], device='cpu')
    y_true = torch.tensor([0, 0, 1, 1], device='cpu')
    with config_context(array_api_dispatch=False):
        average_precision_score(y_true, y_scores)
```

Thus to make testing easier, I checked the above on main (works fine) and on this branch with the above lines commented out (works as well).

From what I can follow, the inputs are converted to numpy arrays?
`average_precision_score` calls `precision_recall_curve` calls `confusion_matrix_at_thresholds` which calls `column_or_1d` on `y_true` and `y_score` ?

> But doesn't average_precision_score call precision_recall_curve ?

I think the changes made to `y_true` and `y_score` made in `precision_recall_curve` stay local.

> Thus to make testing easier, I checked the above on main (works fine) and on this branch with the above lines commented out (works as well).

Interesting. It fails on the mps CI (but fails on a different test case for a similar reason (at np.repeat) locally for me on linux).

> From what I can follow, the inputs are converted to numpy arrays?
average_precision_score calls precision_recall_curve calls confusion_matrix_at_thresholds which calls column_or_1d on y_true and y_score ?

Let me actually comment out this patch and push, so you can inspect on CI.

Here it is: #33028

I have opened a new PR for this to have the CI run and also to explore alternative solutions there.

Do we need this `if` statement? `array_api_strict` is super strict and sticks as close as possible to what the array API standard mandates. No compromises and no historical baggage.

For me this means we should not special case it, instead we should write our code so that it works with `array_api_strict` and only add workarounds for libraries that are not (strictly) compliant with the standard.

Is that view too strict/not pragmatic enough?

I wanted to prevent adding a permanent fix on a seemingly temporary problem.

We have a history of type compatibilities between `float64` and `int64` in `array_api_strict` (https://github.com/[USER_REDACTED]n/scikit-learn/pull/32422#issuecomment-3605073399, #32552, #33022 being the lastes ones, I think) getting fixed with newer scipy releases it seems.

(This one actually fails with scipy==1.16.3 for me locally but I didn't go into depth.)

Just curious (and I couldn't find an answer in #32552) - why is it only array api strict that fails (and not other array inputs)?
My initial understanding was that it was a scipy 'bug' meaning that certain functions were not able to handle mixed dtypes (but if that was the case, why does it not affect other array libraries?), but is it actually (also?) related to some array method?

I am not sure, to be honest, and I don't remember.

I agree, this smells bad, I wouldn't feel comfortable with this condition. Code should be compatible with array api strict. So we need to dig deep and figure out the issue here.

I'd say if the issue is type incompatibility, we should simply check for types and convert them before the operation. Kinda where it's needed to avoid extra overhead.

seems like `c` needs to be on the right device already, which also avoids repeating the move twice here.

If it's an integer, why can't we just index, if it's a list, it should be on the right device, or have a version on the right device, since it seems `score[c]` is fine?

Otherwise, it seems we need a sort of a `_safe_indexing` which can handle device difference between the index and the values. But I don't like having to move indices every time we index.

I agree we should comply with the spec, if this is not a bug in `array-api-strict`.

I went down to investigate this:

The error comes from `array_api_strict.multiply` where `y_true` has dtype `array_api_strict.int64` and `xp.reshape(score_weight, (-1, 1))` has dtype `array_api_strict.float64`.

`xp.reshape` must follow the "Type Promotion Rules", see: https://data-apis.org/array-api/latest/API_specification/generated/array_api.multiply.html#multiply

The [Type Promotion Rules](https://data-apis.org/array-api/latest/API_specification/type_promotion.html#type-promotion) in turn state that "mixed-kind promotion is undefined" (though I wonder what where the reasons to design it like this). data-apis/array-api-strict#136 shows that the type incompatibility is indeed very on purpose.

This means we need to do the type promotion in our code without special casing it to the `array_api_strict` namespace.

I have changed the code accordingly.

`c` is an int.

> If it's an integer, why can't we just index,

This would work, [USER_REDACTED]:

```diff
-        y_true_c = _ravel(
-            xp.take(y_true, xp.asarray([c], device=_device), axis=not_average_axis)
-         )
-        y_score_c = _ravel(
-            xp.take(y_score, xp.asarray([c], device=_device), axis=not_average_axis)
-        )
+        y_true_rotated = xp.moveaxis(y_true, not_average_axis, -1)
+        y_true_c = _ravel(y_true_rotated[..., c])
+        y_score_rotated = xp.moveaxis(y_score, not_average_axis, -1)
+        y_score_c = _ravel(y_score_rotated[..., c])
```

Though I didn't understand _why_ we want to change this. What is the advantage of doing it differently?

I have changed this code according to my suggestion in https://github.com/[USER_REDACTED]n/scikit-learn/pull/33028#issuecomment-3749462746.

Please let me know if that's not a good solution in your opinion. Happy to change it.

Double checking https://data-apis.org/array-api/latest/API_specification/generated/array_api.take.html

Now I see `index` and `x` need to be both an `array`, which means we have to create an array anyway, which means at the point we do that, makes sense for them to be on the same device. Still, makes odd code.

> xp.moveaxis(y_score, not_average_axis, -1)

Since this operation can be quite expensive, we probably shouldn't do it anyway.

We can keep the code as is, still, odd.

```suggestion
        return float(_average(score, weights=average_weight, xp=xp))
```

Ah interesting. So does this mean that `multiply` for torch and cupy (the other libraries we test) don't fail because they do allow mixed type promotion?

So is this unrelated then to issue #32552 , and thus would not be fixed by [NAME_REDACTED]?

> Ah interesting. So does this mean that multiply for torch and cupy (the other libraries we test) don't fail because they do allow mixed type promotion?

Yes, it seems so. Numpy does, too.

> So is this unrelated then to issue https://github.com/[USER_REDACTED]n/scikit-learn/issues/32552 , and thus would not be fixed by [NAME_REDACTED]?

I am not sure, but it seems unrelated. Maybe also I misread the cause of https://github.com/[USER_REDACTED]n/scikit-learn/issues/32552.

Thanks for reviewing, [USER_REDACTED]!

I have now added `xp`, as you suggested, though it gets overridden in the first line of `_average` anyway.
I had always wondered why we extract `xp` anew for helper functions, because whatever gets passed as `xp` gets reassigned then, but here, I agree it is good to add `xp=xp` for consistency and making the intend clear.

I think `get_namespace_and_device` gets passed `xp`, and it only gets namespace if `xp=None`

```suggestion
        # we need y_pred_nd to be of shape (n_samples, n_classes)
```
nit

```suggestion
            #  Mixed integer and float type promotion not defined in array standard
```

why not
```suggestion
            y_true = xp.asarray(y_true, dtype=score_weight.dtype)
```

I probably missed something by [NAME_REDACTED] `roc_auc_score` ?

Yes, it ready much cleaner, thanks.

`roc_auc_score` shares `_average_binary_score` as a common helper function with `average_binary_score`. With the changes in there, we now get an error there (`AttributeError: 'list' object has no attribute 'dtype'`) because `roc_auc_score`  had previously not handled `sample_weight` correctly either.


DOC Add link to glossary "label indicator matrix" for classification metric docstrings

#### [NAME_REDACTED]/PRs
Noticed while working on #32755

#### What does this implement/fix? Explain your changes.
Adds reference to the glossary term https://scikit-learn.org/dev/glossary.html#term-label-indicator-matrix for classification metrics

Not sure about the best term to use.

* Avoided use of 'binary' i.e., binary indicator - binary here is talking about the indicator being 0 or 1, not about the data being binary. This *may* be confusing and this term is not included as an alias in the glossary so I have avoided it
* when talking about binary data, I have used the term "label indicator format“ / ”label indicator matrix“ (depending on how best the wording fits into the sentence), to avoid use of 'multilabel' (as the data is not multilabel).
* used 'multilabel indicator matrix' when talking about multilabel data

#### AI usage disclosure
<!--
If AI tools were involved in creating this PR, please check all boxes that apply
below and make sure that you adhere to our [NAME_REDACTED]:
https://scikit-learn.org/dev/developers/contributing.html#automated-contributions-policy
-->
I used AI assistance for:
- [ ] Code generation (e.g., when writing an implementation or fixing a bug)
- [ ] Test/benchmark generation
- [ ] Documentation (including examples)
- [ ] Research and understanding

#### Any other comments?

<!--
Thank you for your patience. Changes to scikit-learn require careful
attention, but with limited maintainer time, not every contribution can be reviewed
quickly.
For more information and tips on improving your pull request, see:
https://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

@@ -518,14 +518,18 @@ [NAME_REDACTED]
         :term:`memory mapping`. See :ref:`parallelism` for more
         information.

+    label indicator format
     label indicator matrix
     multilabel indicator matrix
     multilabel indicator matrices
-        The format used to represent multilabel data, where each row of a 2d
-        array or sparse matrix corresponds to a sample, each column
+        This format can be used to represent binary or multilabel data. Each row of
+        a 2d array or sparse matrix corresponds to a sample, each column
         corresponds to a class, and each element is 1 if the sample is labeled
         with the class and 0 if not.

+        :ref:`LabelBinarizer <preprocessing_targets>` can be used to create a
+        multilabel indicator matrix from :term:`multiclass` labels.
+
     leakage
     data leakage
         A problem in cross validation where generalization performance can be

@@ -142,7 +142,7 @@ def average_precision_score(
     Parameters
     ----------
     y_true : array-like of shape (n_samples,) or (n_samples, n_classes)
-        True binary labels or binary label indicators.
+        True binary labels or :term:`multilabel indicator matrix`.

     y_score : array-like of shape (n_samples,) or (n_samples, n_classes)
         Target scores, can either be probability estimates of the positive
@@ -502,9 +502,9 @@ def roc_auc_score(
     Parameters
     ----------
     y_true : array-like of shape (n_samples,) or (n_samples, n_classes)
-        True labels or binary label indicators. The binary and multiclass cases
+        True labels or :term:`label indicator matrix`. The binary and multiclass cases
         expect labels with shape (n_samples,) while the multilabel case expects
-        binary label indicators with shape (n_samples, n_classes).
+        a :term:`multilabel indicator matrix` with shape (n_samples, n_classes).

     y_score : array-like of shape (n_samples,) or (n_samples, n_classes)
         Target scores.
@@ -1337,7 +1337,7 @@ def label_ranking_average_precision_score(y_true, y_score, *, sample_weight=None
     Parameters
     ----------
     y_true : {array-like, sparse matrix} of shape (n_samples, n_labels)
-        True binary labels in binary indicator format.
+        True binary labels in :term:`label indicator format`.

     y_score : array-like of shape (n_samples, n_labels)
         Target scores, can either be probability estimates of the positive
@@ -1439,7 +1439,7 @@ def coverage_error(y_true, y_score, *, sample_weight=None):
     Parameters
     ----------
     y_true : array-like of shape (n_samples, n_labels)
-        True binary labels in binary indicator format.
+        True binary labels in :term:`label indicator format`.

     y_score : array-like of shape (n_samples, n_labels)
         Target scores, can either be probability estimates of the positive
@@ -1516,7 +1516,7 @@ def label_ranking_loss(y_true, y_score, *, sample_weight=None):
     Parameters
     ----------
     y_true : {array-like, sparse matrix} of shape (n_samples, n_labels)
-        True binary labels in binary indicator format.
+        True binary labels in :term:`label indicator format`.

     y_score : array-like of shape (n_samples, n_labels)
         Target scores, can either be probability estimates of the positive

Thanks for the PR, [USER_REDACTED]. I find these kind of doc improvements very useful.
I only have a nit comment and a suggestion to your question.

Thank you for the PR! LGTM

I've used both glossary aliases 'label indicator matrix' (when talking about binary data) and 'multilabel indicator matrix' (when talking about multilabel data). I think this is okay. I amended the glossary entry to clarify that it could be used for both binary and multilabel data...

Also wonder if we should add a short example, like we do here: https://scikit-learn.org/dev/modules/preprocessing_targets.html#multilabelbinarizer

What do you think about adding:

```suggestion
        with the class and 0 if not. :class:`LabelBinarizer` creates a
        :term:`multilabel indicator matrix` from :term:`multiclass` labels.
```

We have a similar sentence in [preprocessing_targets.rst](https://scikit-learn.org/stable/modules/preprocessing_targets.html#labelbinarizer) followed be an example, but the link to `LabelBinarizer` is probably enough.

```suggestion
        a :term:`multilabel indicator matrix` with shape (n_samples, n_classes).
```

Nit.

I changed to reference the user guide `LabelBinarizer` section, and not the `LabelBinarizer` API, as it seemed to fit more.


MNT/DOC Autoclose schedule doesn't run on forks and improved structuring

#### [NAME_REDACTED]/PRs
- fixes https://github.com/[USER_REDACTED]n/scikit-learn/pull/32743#issuecomment-3641069962
- fixes https://github.com/[USER_REDACTED]n/scikit-learn/pull/32660#issuecomment-3546620178
-  implements ideas from discussion with [USER_REDACTED]

#### What does this implement/fix? Explain your changes.
This PR fixes the `autoclose-schedule` workflow to not run on forks, links to a central list of how to improve a PR, and implements some changes in the wording. Specifically, it makes it more clear to new contributors where the autoclose label was set, that they are responsible to fix the issues and advises them to not request the label getting removed.

#### AI usage disclosure
<!--
If AI tools were involved in creating this PR, please check all boxes that apply
below and make sure that you adhere to our [NAME_REDACTED]:
https://scikit-learn.org/dev/developers/contributing.html#automated-contributions-policy
-->
I used AI assistance for:
- [ ] Code generation (e.g., when writing an implementation or fixing a bug)
- [ ] Test/benchmark generation
- [ ] Documentation (including examples)
- [x] Research and understanding

@@ -39,36 +39,29 @@ jobs:

             Thank you for your contribution to scikit-learn and for the effort you have
             put into this PR. This pull request does not yet meet the quality and
-            clarity needed for an effective review. Reviewing time is limited, and our
-            goal is to prioritize well-prepared contributions to keep scikit-learn
-            maintainable. Unless this PR is improved, it will be automatically closed
-            after two weeks.
+            clarity needed for an effective review. Project maintainers have limited
+            time for code reviews, and our goal is to prioritize well-prepared
+            contributions to keep scikit-learn maintainable.

-            To avoid autoclose and increase the chance of a productive review, please:
+            To increase the chance of a productive review, please refer to: [How do I
+            improve my issue or pull
+            request?](https://scikit-learn.org/dev/faq.html#how-do-i-improve-my-issue-or-pull-request)
+            As the author, you are responsible for driving this PR, which entails doing
+            necessary background research as well as presenting its context and your
+            thought process. If you are a [new
+            contributor](https://scikit-learn.org/dev/developers/contributing.html#new-contributors),
+            or do not know how to fulfill these requirements, we recommend that you
+            familiarise yourself with scikit-learn's development conventions via other
+            contribution types (e.g., reviewing PRs) before submitting code.

-            - Ensure your contribution aligns with our
-            [contribution guide](https://scikit-learn.org/dev/developers/contributing.html).

-            - Include a clear motivation and concise explanation in the pull request
-            description of why you chose this solution.
+            Scikit-learn maintainers cannot provide one-to-one guidance on this PR.
+            However, if you ask focused, well-researched questions, a community
+            member may be willing to help. 💬

-            - Make sure the code runs and passes tests locally (`pytest`) and in the CI.

-            - Submit only code you can explain and maintain; reviewers will ask for
-            clarifications and changes. Disclose any AI assistance per our
-            [[NAME_REDACTED]](https://scikit-learn.org/dev/developers/contributing.html#automated-contributions-policy).
-
-            - Keep the changes minimal and directly relevant to the described issue or
-            enhancement.
-
-
-            We cannot provide one-to-one guidance on every PR, though we
-            encourage you to ask focused, actionable questions that show you have tried
-            to explore the problem and are interested to engage with the project. 💬
-            Sometimes a maintainer or someone else from the community might be able to
-            offer pointers.
-
-
-            If you improve your PR within the two-week window, the `autoclose` label can
-            be removed by maintainers.
+            If you substantially improve this PR within two weeks, a team member may
+            remove the `autoclose` label and the PR stays open. Cosmetic changes or
+            incomplete fixes will not be sufficient. Maintainers will assess
+            improvements on their own schedule. Please do not ping (`@`) maintainers.

@@ -18,6 +18,7 @@ jobs:
   autoclose:
     name: autoclose labeled PRs
     runs-on: ubuntu-latest
+    if: github.repository == 'scikit-learn/scikit-learn'
     steps:
       - uses: actions/checkout@v5
       - uses: actions/setup-python@v6

@@ -33,14 +33,16 @@ jobs:
           /repos/$GH_REPO/issues/$ISSUE_NUMBER/comments \
           -f "body=$BODY"
         env:
-          BODY: |
+          BODY: >
             Thanks for the work you've done so far. The goal of this comment
             is to set expectations.

+
             Deciding on new features or substantial changes is a lengthy
             process. It frequently happens that no maintainer is available
             to take on this task right now.

+
             Please do not create a Pull Request before a decision has been
             made regarding the proposed work. Making this decision can
             often take a significant amount of time and effort.

@@ -37,8 +37,7 @@ See :ref:`ways_to_contribute` to learn how to make meaningful contributions.

 .. topic:: **Our community, our values**

-    We are a community based on openness and friendly, didactic,
-    discussions.
+    We are a community based on openness and friendly, didactic discussions.

     We aspire to treat everybody equally, and value their contributions.  We
     are particularly seeking people from underrepresented backgrounds in Open

@@ -313,17 +313,18 @@ being reviewed, you can try:

 * follow our :ref:`contribution guidelines <contributing>`, in particular
   :ref:`automated_contributions_policy`, :ref:`filing_bugs`,
-  :ref:`stalled_pull_request` and :ref:`stalled_unclaimed_issues`.
-* complete the provided issue and pull request templates, including a clear and
-  concise description of the issue or motivation for the pull request.
+  :ref:`stalled_pull_request` and :ref:`stalled_unclaimed_issues`,
+* complete all sections of the issue or pull request template provided by GitHub,
+  including a clear description of the issue or motivation and thought process behind
+  the pull request
 * ensure the title clearly describes the issue or pull request and does not include
   an issue number.

 For your pull requests specifically, the following will make it easier to review:

-* ensure your PR satisfies all items in the
-  :ref:`Pull request checklist <pr_checklist>`.
-* ensure your PR addresses an issue for which there is clear consensus on the solution.
+* ensure your PR addresses an issue for which there is clear consensus on the solution
+  (see :ref:`issues_tagged_needs_triage`),
+* ensure the PR satisfies all items in the :ref:`Pull request checklist <pr_checklist>`,
 * ensure the changes are minimal and directly relevant to the described issue.

 What does the "spam" label for issues or pull requests mean?

Overall I think we can be more succinct and structured in the autoclose comment. I am not sure repeating ourselves is going to stop the more egregious offenders and the longer we make this, the less likely people are going to read it.

What about an overall structure of:

* Paragraph 1: PR does not standards (I think it's fine currently)
* Paragraph 2: Link to ways to improve PR. Advise they are responsible for driving this PR. If they do not know how to improve their PR, suggest building foundation with other contribution types (link to new contributor section) or asking questions after doing research, which someone from community may respond to.
* Paragraph 3: If they improve the PR, the label may be removed. Advise against them requesting label be removed or ping-ing maintainers (I *think* you did not want to tell people not to ping maintainers, but I think its worth having, they can still comment and ask questions, but don't ping maintainers)

I've made more specific suggestions below.

Thanks for your feedback, [USER_REDACTED], which is valuable and I have implemented most of the requested changes, except in the paragraph that keeps the door open for people who want to learn where I hope we can resolve this together.

Though I can understand your concern about repeating ourselves, this PR doesn't make the comment longer and cutting it down to only three paragraphs you suggested feel a bit strict to me. My goal also is to help us feel less guilty when using this label and for me it helps to be a bit more generous with information, since I have unexperienced but genuine people in mind that read this (knowing that authors of the careless fly-by [NAME_REDACTED], but who cares). I have tried not to speak of "us" but instead of "scikit-learn", because the former sounds gatekeepy, and I don't want to spell out something like  "build your foundations first" (which I find degrading) but instead to talk about familiarising yourself with the repo.

I have added "please do not ping maintainers", after experiencing some pings on incomplete PRs myself, I think its necessary.

I think it reads much better, but personally still prefer conciseness (but this is more of a personal preference).

Some thoughts but would be good to get someone else's perspective as well.

Thank you! LGTM

Thanks for this PR! Let's merge this one and iterate on it if we need

I think this is already mentioned in the PR checklist

I though the original was mostly adequate? I am reluctant to make it wordier, but may we could add that they should explain their through process for PRs...
```suggestion
* complete all sections of the issue or pull request template,
  including a clear and concise description of the issue or motivation and
  thought process behind the pull request
```

I think this may be included in the AI contributions section and honestly I am not sure writing this several times is not going to stop offending users. These people are unlikely to read or abide by [NAME_REDACTED] many times we write it...

I feel like this is a bit vague and I think the PR checklist already does a reasonable job of discussing some of these conventions: https://scikit-learn.org/dev/developers/contributing.html#pull-request-checklist

Not sure about how this phrase comes off: "if all that sounds demanding, that's because it is,"

This seems vague - helpful to whom? And I am worried someone will use this against us, because we don't have time to engage in discussions.

Good to get your feedback on this. I was thinking about this a lot. My intention was to show a bit of appreciation for the work they would have to put in, but I can see it is the wrong place to do that.

I think that too often we express "why can't you simply do that easy thing" without appreciating that it is not a little thing to ask people to do.

I agree it is not fitting this context, so I have deleted this.

I am referring to this section in our values: "We are a community based on openness and friendly, didactic discussions."

This sentence tries to keep the door open to junior people, who are interested to learn. I am actually pretty fond of expressing something like this. I feel the autoclose message is too harsh otherwise.

Also, I would like to address a broad range of contributions with this label. Right now I feel only really careless PRs would deserve it, and I would not put it on a PR that lacks quality, but the contributors seems to have spend a lot of time getting it right.

> And I am worried someone will use this against us, because we don't have time to engage in discussions.

I understand your concerns and I am very sure that some people will misuse this. That's why I tried to express some conditions following this sentence:
- investigate your problem yourself first
- be respectful (not pushing generated comments)
- answer is a gift from the community and not guaranteed
- answer may come from the community, not necessarily maintainers

What do you think about encouraging community discussions without promising guidance and if you think it is worthwhile: How could we express that so it doesn't get misused?

Yes, that exactly was my motivation, since we had talked about it before. Also, I was not sure if new contributors would know what "issue and pull request templates" where and that's why I added "provided by GitHub". I have now updated this bullen point with your suggestion and added "provided by GitHub", just in case.

Sadly true. I think the AI contributions section already does a great job explaining this, so I deleted this here.

True, I have removed it from here.

I can now see it is vague, since what I had meant is not covered in the Pull Request Checklist. My intention was to give new contributors a bit of a hand on how to work if they want to make their PR fit in. Like, check other PRs similar to this one, then you can find out things like: the correct location to put a new test, that you need param validation if you add a new param, that we have utility functions that you can use, that we never add new dependencies just for a fix, that there will be a lot of discussion and you will have to keep working on this for a while, and so on. It is the same as telling people to familiarise themselves with the repo first before attempting to add a new feature.

Maybe this is a no-brainer anyways. Since my attempt to express this was too vague, I have removed this here.

Maybe?
```suggestion
            improvements on their own schedule. Please do not ping (`@`) maintainers.
```

```suggestion
            familiarise yourself with scikit-learn's development conventions via other
```

> Also, I would like to address a broad range of contributions with this label.

Ah, I was mostly thinking of the most egregious contributions, as that is what I have been using the label for. I'd like to think that for a contribution that looks like someone personally put effort in, I'd give pointers instead - but maybe we don't have time for this. Is the idea that it is better that these contributors get some response instead of nothing? I think I'd have to see some of these contributions in action to make a judgement here.

> a community member (most likely not a maintainer) might be
            willing to offer advice.

Just curious, does this happen often? Don't feel strongly about this section one way or another, but I am just worried they won't get much response.

Ah that makes sense. I can see what you mean, this stuff comes with experience.

I do wonder if these are the main problems with the PRs deserving of this label. e.g., if it's an otherwise well thought out PR, asking them to add param validation would not be a big deal for me. Similarly, I would have thought that PRs that address an issue where a clear path forward has been decided would not result in adding an unwarranted new dependency.

Thinking about this more, I am not sure about this section - with respect to your sentiment of not wanting to sound too harsh, and the idea that small issues in an otherwise well prepared PR is probably not the main problem.

I like the idea of saying that they are responsible for driving this PR, but the long list may seem intimidating to genuine contributors. You'd also think genuine contributors would already answer questions and address maintainer suggestions. Fixing test failures are already mentioned in the PR checklist.

For the use cases I have seen, I feel like the problem is more the lack of background research, lack of context provided and no description of their though process/why they decided on this solution/what alternatives were thought about. Of course, some PRs also give the bare minimum (no tests, no doc changes, CI may or may not be failing) and call it 'fixing' an issue.

I find it sometimes hard to judge how much effort was put into something and I feel more at ease when we express something like "Sorry we (maintainers) cannot guide you, but there is a whole community and maybe someone else will."

> Just curious, does this happen often?

Currently, rarely. But I'd really like to see it happen. Writing this hope down might help this happening.

At the same time since #32734 we guide new contributors towards starting by [NAME_REDACTED] code, so this combination makes sense to me.

I understand what you mean. I have changed this part of the message into the following:

```
As the author, you are responsible for driving this PR, which entails
doing necessary background research as well as presenting its context and
your thought process.
```

Thinking about the phrase I wrote above ("Sorry we (maintainers) cannot guide you, but there is a whole community and maybe someone else will.") I think a friendly variation of this might be enough. The important thing here is to refer the discussion to the community, and we don't need to educate people on how to not be offensive / demanding / pretentious when communicating online.

So I've significantly shortened this paragraph to:

```
Scikit-learn maintainers cannot provide one-to-one guidance on this PR.
However, if you ask focused, well-researched questions, a community
member may be willing to help. 💬
```

This sounds good!

> At the same time since https://github.com/[USER_REDACTED]n/scikit-learn/pull/32734 we guide new contributors towards starting by [NAME_REDACTED] code, so this combination makes sense to me.

I agree and this is a nice approach/direction!

This is nice!

```suggestion
            or do not know how to fulfill these requirements, we recommend that
            you familiarise yourself with scikit-learn's development conventions via other
```
optional change, just thought this reads a bit nicer


ENH add clearer error message when instance is passed instead of class

<!--
🙌 Thanks for contributing a pull request!

👀 Please ensure you have taken a look at the contribution guidelines:
https://github.com/[USER_REDACTED]n/scikit-learn/blob/main/CONTRIBUTING.md

✅ In particular following the pull request checklist will increase the likelihood
of having maintainers review your PR:
https://scikit-learn.org/dev/developers/contributing.html#pull-request-checklist

📋 If your PR is likely to affect users, you will need to add a changelog entry
describing your PR changes, see:
https://github.com/[USER_REDACTED]n/scikit-learn/blob/main/doc/whats_new/upcoming_changes/README.md
-->

#### [NAME_REDACTED]/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes https://github.com/[USER_REDACTED]n/scikit-learn/issues/32719

#### What does this implement/fix? Explain your changes.
It adds a simple check to `Pipeline`, `FeatureUnion` and `ColumnTransformer` to raise a clearer error message if the user accidentally passed a class instead of an instance, and provides corresponding non-regression test.

#### AI usage disclosure
<!--
If AI tools were involved in creating this PR, please check all boxes that apply
below and make sure that you adhere to our [NAME_REDACTED]:
https://scikit-learn.org/dev/developers/contributing.html#automated-contributions-policy
-->
I used AI assistance for:
- [ ] Code generation (e.g., when writing an implementation or fixing a bug)
- [ ] Test/benchmark generation
- [ ] Documentation (including examples)
- [ ] Research and understanding

#### Any other comments?
As noted in my comment [here](https://github.com/[USER_REDACTED]n/scikit-learn/issues/32719#issuecomment-3638023309), it could also be added to the existing `_validate_estimator()` function in `_BaseHeterogeneousEnsemble` and its version in `StackingClassifier` in a follow-up PR. If I followed the class hierarchy correctly, then this would also catch this mistake in `StackingClassifier`, `StackingRegressor`, `VotingClassifier` and `VotingRegressor`. I didn't check the other meta-estimators yet.

<!--
Thank you for your patience. Changes to scikit-learn require careful
attention, but with limited maintainer time, not every contribution can be reviewed
quickly.
For more information and tips on improving your pull request, see:
https://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

@@ -0,0 +1,4 @@
+- :class:`pipeline.Pipeline`, :class:`pipeline.FeatureUnion` and
+  :class:`compose.ColumnTransformer` now raise a clearer
+  error message when an estimator class is passed instead of an instance.
+  By :user:`[NAME_REDACTED] <AnneBeyer>`

@@ -513,6 +513,7 @@ def _validate_transformers(self):
         self._validate_names(names)

         # validate estimators
+        self._check_estimators_are_instances(transformers)
         for t in transformers:
             if t in ("drop", "passthrough"):
                 continue

@@ -93,6 +93,23 @@ def transform(self, X, y=None):
         raise ValueError("specific message")

+[USER_REDACTED].mark.parametrize(
+    "transformers",
+    [
+        [("trans1", Trans, [0]), ("trans2", Trans(), [1])],
+        [("trans1", Trans(), [0]), ("trans2", Trans, [1])],
+        [("drop", "drop", [0]), ("trans2", Trans, [1])],
+        [("trans1", Trans, [0]), ("passthrough", "passthrough", [1])],
+    ],
+)
+def test_column_transformer_raises_class_not_instance_error(transformers):
+    # non-regression tests for https://github.com/[USER_REDACTED]n/scikit-learn/issues/32719
+    ct = ColumnTransformer(transformers)
+    msg = "Expected an estimator instance (.*()), got estimator class instead (.*)."
+    with pytest.raises(TypeError, match=msg):
+        ct.fit([[1]])
+
+
 def test_column_transformer():
     X_array = np.array([[0, 1, 2], [2, 4, 6]]).T

@@ -296,6 +296,7 @@ def _validate_steps(self):
         self._validate_names(names)

         # validate estimators
+        self._check_estimators_are_instances(estimators)
         transformers = estimators[:-1]
         estimator = estimators[-1]

@@ -1698,6 +1699,7 @@ def _validate_transformers(self):
         self._validate_names(names)

         # validate estimators
+        self._check_estimators_are_instances(transformers)
         for t in transformers:
             if t in ("drop", "passthrough"):
                 continue

@@ -282,6 +282,27 @@ def test_pipeline_invalid_parameters():
     assert params == params2

+[USER_REDACTED].mark.parametrize(
+    "meta_estimators",
+    [
+        Pipeline([("pca", PCA)]),
+        Pipeline([("pca", PCA), ("ident", None)]),
+        Pipeline([("passthrough", "passthrough"), ("pca", PCA)]),
+        Pipeline([("passthrough", None), ("pca", PCA)]),
+        Pipeline([("scale", StandardScaler), ("pca", PCA())]),
+        FeatureUnion([("pca", PCA), ("svd", TruncatedSVD())]),
+        FeatureUnion([("pca", PCA()), ("svd", TruncatedSVD)]),
+        FeatureUnion([("drop", "drop"), ("svd", TruncatedSVD)]),
+        FeatureUnion([("pca", PCA), ("passthrough", "passthrough")]),
+    ],
+)
+def test_meta_estimator_raises_class_not_instance_error(meta_estimators):
+    # non-regression tests for https://github.com/[USER_REDACTED]n/scikit-learn/issues/32719
+    msg = "Expected an estimator instance (.*()), got estimator class instead (.*)."
+    with pytest.raises(TypeError, match=msg):
+        meta_estimators.fit([[1]])
+
+
 def test_empty_pipeline():
     X = iris.data
     y = iris.target

@@ -100,6 +100,14 @@ def _validate_names(self, names):
                 "Estimator names must not contain __: got {0!r}".format(invalid_names)
             )

+    def _check_estimators_are_instances(self, estimators):
+        for estimator in estimators:
+            if isinstance(estimator, type):
+                raise TypeError(
+                    "Expected an estimator instance ({estimator.__name__}()), got "
+                    "estimator class instead ({estimator.__name__})."
+                )
+

 def _safe_split(estimator, X, y, indices, train_indices=None):
     """Create subset of dataset and properly handle kernels.

This is pretty neat. LGTM, but it can potentially be in our input validation. I'd need [USER_REDACTED] 's opinion on this though.

lgtm, thanks

```suggestion
  :class:`compose.ColumnTransformer` now raise a clearer
```

we should always refer to the public API in the changelog.


FIX: Tooltip position using CSS anchor positioning

#### [NAME_REDACTED]/PRs

The documentation tooltip that we introduced with https://github.com/[USER_REDACTED]n/scikit-learn/pull/31564 is hidden if the parameter name that it belongs to, is at the bottom of the table. It's visible only after scrolling down the cell.
In this case, when we hover over `n_jobs`, we get:

Using [CSS anchor positioning:](https://developer.mozilla.org/en-US/docs/Web/CSS/Guides/Anchor_positioning/Using), the result is the following (when hovering over `n_jobs`). No scroll-down needed:

There is no opened issue for this.

#### What does this implement/fix? Explain your changes.
For the record: The specs for `anchor-name` and `position-anchor` can't be placed in the `params.css` file. When I did that, the tooltip was always visible at the bottom of the table - not next to the parameter name.
(It was [USER_REDACTED]'s idea that the tooltip was getting fixed to the last parameter when using Chrome.)

#### AI usage disclosure

I used AI assistance for:
- [x] Moving the style to `params.py` instead of keeping it on `params.css`.
- [ ] Test/benchmark generation
- [ ] Documentation (including examples)
- [ ] Research and understanding

#### Any other comments?
I tried to set [Fallback options](https://developer.mozilla.org/en-US/docs/Web/CSS/Guides/Anchor_positioning/Try_options_hiding) ~~but I haven't found the correct rule.~~

Edit: I think the UX would be annoying if I add the fallback options.

@@ -0,0 +1,6 @@
+- The parameter table in the HTML representation of all scikit-learn
+  estimators inheritiging from :class:`base.BaseEstimator`, displays
+  each parameter documentation as a tooltip. The last tooltip of a
+  parameter in the last table of any HTML representation was partially hidden.
+  This issue has been fixed.
+  By :user:`Dea María Léon <DeaMariaLeon>`

@@ -83,6 +83,14 @@ a.param-doc-link:visited {
     padding: .5em;
 }

+[USER_REDACTED](anchor-name: --doc-link) {
+    a.param-doc-link,
+    a.param-doc-link:link,
+    a.param-doc-link:visited {
+    anchor-name: --doc-link;
+    }
+}
+
 /* "hack" to make the entire area of the cell containing the link clickable */
 a.param-doc-link::before {
     position: absolute;
@@ -109,6 +117,14 @@ a.param-doc-link::before {
     border: thin solid var(--sklearn-color-unfitted-level-3);
 }

+[USER_REDACTED](position-area: center right) {
+    .param-doc-description {
+    position-area: center right;
+    position: fixed;
+    margin-left: 0;
+    }
+}
+
 /* Fitted state for parameter tooltips */
 .fitted .param-doc-description {
     /* fitted */

@@ -89,9 +89,12 @@ def _params_html_repr(params):

     PARAM_AVAILABLE_DOC_LINK_TEMPLATE = """

             {param_name}
-            {param_description}
+
+            {param_description}

     """
     estimator_class_docs = inspect.getdoc(params.estimator_class)

Otherwise, it is an improvement. LGTM.

Looking at the `anchor-name` [support table](https://developer.mozilla.org/en-US/docs/Web/CSS/Reference/Properties/anchor-name#browser_compatibility) it is not supported in Firefox yet.

With this change, what does the tooltip look like in Firefox?

This is some remaining debugging I think.

🫣 Thanks


TST Add `confusion_matrix_at_thresholds` to common tests

#### [NAME_REDACTED]/PRs

#### What does this implement/fix? Explain your changes.
`confusion_matrix_at_thresholds` was amended to a public function added in #30134. I think it would be worth adding to the common metric tests in `test_common.py`.
This would also fix one of the test failures in #32755

Also fixes `test_binary_sample_weight_invariance` - the curve metrics, as well as `CONTINUOUS_CLASSIFICATION_METRICS`, should also take `y_score` (i.e. unthresholded scores) NOT `y_pred`

#### AI usage disclosure
<!--
If AI tools were involved in creating this PR, please check all boxes that apply
below and make sure that you adhere to our [NAME_REDACTED]:
https://scikit-learn.org/dev/developers/contributing.html#automated-contributions-policy
-->
I used AI assistance for:
- [ ] Code generation (e.g., when writing an implementation or fixing a bug)
- [ ] Test/benchmark generation
- [ ] Documentation (including examples)
- [x] Research and understanding

#### Any other comments?

cc [USER_REDACTED] because this is related to #32755 and [USER_REDACTED] and [USER_REDACTED]  who reviewed #30134

@@ -18,6 +18,7 @@
     classification_report,
     cohen_kappa_score,
     confusion_matrix,
+    confusion_matrix_at_thresholds,
     coverage_error,
     d2_absolute_error_score,
     d2_brier_score,
@@ -156,17 +157,13 @@
     "balanced_accuracy_score": balanced_accuracy_score,
     "adjusted_balanced_accuracy_score": partial(balanced_accuracy_score, adjusted=True),
     "unnormalized_accuracy_score": partial(accuracy_score, normalize=False),
-    # `confusion_matrix` returns absolute values and hence behaves unnormalized
-    # . Naming it with an unnormalized_ prefix is necessary for this module to
-    # skip sample_weight scaling checks which will fail for unnormalized
-    # metrics.
-    "unnormalized_confusion_matrix": confusion_matrix,
+    "confusion_matrix": confusion_matrix,
     "normalized_confusion_matrix": lambda *args, **kwargs: (
         confusion_matrix(*args, **kwargs).astype("float")
         / confusion_matrix(*args, **kwargs).sum(axis=1)[:, np.newaxis]
     ),
-    "unnormalized_multilabel_confusion_matrix": multilabel_confusion_matrix,
-    "unnormalized_multilabel_confusion_matrix_sample": partial(
+    "multilabel_confusion_matrix": multilabel_confusion_matrix,
+    "multilabel_confusion_matrix_sample": partial(
         multilabel_confusion_matrix, samplewise=True
     ),
     "hamming_loss": hamming_loss,
@@ -240,6 +237,7 @@ def precision_recall_curve_padded_thresholds(*args, **kwargs):

 CURVE_METRICS = {
+    "confusion_matrix_at_thresholds": confusion_matrix_at_thresholds,
     "roc_curve": roc_curve,
     "precision_recall_curve": precision_recall_curve_padded_thresholds,
     "det_curve": det_curve,
@@ -305,7 +303,7 @@ def precision_recall_curve_padded_thresholds(*args, **kwargs):
     "samples_recall_score",
     "samples_jaccard_score",
     "coverage_error",
-    "unnormalized_multilabel_confusion_matrix_sample",
+    "multilabel_confusion_matrix_sample",
     "label_ranking_loss",
     "label_ranking_average_precision_score",
     "dcg_score",
@@ -327,6 +325,7 @@ def precision_recall_curve_padded_thresholds(*args, **kwargs):
     "f2_score",
     "f0.5_score",
     # curves
+    "confusion_matrix_at_thresholds",
     "roc_curve",
     "precision_recall_curve",
     "det_curve",
@@ -356,6 +355,7 @@ def precision_recall_curve_padded_thresholds(*args, **kwargs):

 # Metrics with a "pos_label" argument
 METRICS_WITH_POS_LABEL = {
+    "confusion_matrix_at_thresholds",
     "roc_curve",
     "precision_recall_curve",
     "det_curve",
@@ -377,7 +377,7 @@ def precision_recall_curve_padded_thresholds(*args, **kwargs):
 # TODO: Handle multi_class metrics that has a labels argument as well as a
 # decision function argument. e.g hinge_loss
 METRICS_WITH_LABELS = {
-    "unnormalized_confusion_matrix",
+    "confusion_matrix",
     "normalized_confusion_matrix",
     "roc_curve",
     "precision_recall_curve",
@@ -406,8 +406,8 @@ def precision_recall_curve_padded_thresholds(*args, **kwargs):
     "macro_precision_score",
     "macro_recall_score",
     "macro_jaccard_score",
-    "unnormalized_multilabel_confusion_matrix",
-    "unnormalized_multilabel_confusion_matrix_sample",
+    "multilabel_confusion_matrix",
+    "multilabel_confusion_matrix_sample",
     "cohen_kappa_score",
     "log_loss",
     "d2_log_loss_score",
@@ -470,7 +470,7 @@ def precision_recall_curve_padded_thresholds(*args, **kwargs):
     "micro_precision_score",
     "micro_recall_score",
     "micro_jaccard_score",
-    "unnormalized_multilabel_confusion_matrix",
+    "multilabel_confusion_matrix",
     "samples_f0.5_score",
     "samples_f1_score",
     "samples_f2_score",
@@ -538,8 +538,9 @@ def precision_recall_curve_padded_thresholds(*args, **kwargs):
     "adjusted_balanced_accuracy_score",
     "explained_variance_score",
     "r2_score",
-    "unnormalized_confusion_matrix",
+    "confusion_matrix",
     "normalized_confusion_matrix",
+    "confusion_matrix_at_thresholds",
     "roc_curve",
     "precision_recall_curve",
     "det_curve",
@@ -552,7 +553,7 @@ def precision_recall_curve_padded_thresholds(*args, **kwargs):
     "weighted_f2_score",
     "weighted_precision_score",
     "weighted_jaccard_score",
-    "unnormalized_multilabel_confusion_matrix",
+    "multilabel_confusion_matrix",
     "macro_f0.5_score",
     "macro_f2_score",
     "macro_precision_score",
@@ -575,6 +576,19 @@ def precision_recall_curve_padded_thresholds(*args, **kwargs):
     "weighted_ovo_roc_auc",
 }

+WEIGHT_SCALE_DEPENDENT_METRICS = {
+    # 'confusion_matrix' metrics returns absolute `tps`, `fps` etc values, which
+    # are scaled by weights, so will vary e.g., scaling by 3 will result in 3 * `tps`
+    "confusion_matrix",
+    "confusion_matrix_at_thresholds",
+    "multilabel_confusion_matrix",
+    "multilabel_confusion_matrix_sample",
+    # Metrics where we set `normalize=False`
+    "unnormalized_accuracy_score",
+    "unnormalized_zero_one_loss",
+    "unnormalized_log_loss",
+}
+
 METRICS_REQUIRE_POSITIVE_Y = {
     "mean_poisson_deviance",
     "mean_gamma_deviance",
@@ -1612,12 +1626,15 @@ def check_sample_weight_invariance(name, metric, y1, y2, sample_weight=None):
         % (weighted_score_zeroed, weighted_score_subset, name),
     )

-    if not name.startswith("unnormalized"):
-        # check that the score is invariant under scaling of the weights by a
-        # common factor
-        # Due to numerical instability of floating points in `cumulative_sum` in
-        # `median_absolute_error`, it is not always equivalent when scaling by a float.
-        scaling_values = [2] if name == "median_absolute_error" else [2, 0.3]
+    # Check the score is invariant under scaling of weights by a constant factor
+    if name not in WEIGHT_SCALE_DEPENDENT_METRICS:
+        # Numerical instability of floating points in `cumulative_sum` in
+        # `median_absolute_error`, and in `diff` when in calculating collinear points
+        # and points in between to drop `roc_curve` means they are not always
+        # equivalent when scaling by a float.
+        scaling_values = (
+            [2] if name in {"median_absolute_error", "roc_curve"} else [2, 0.3]
+        )
         for scaling in scaling_values:
             assert_allclose(
                 weighted_score,
@@ -1715,7 +1732,7 @@ def test_binary_sample_weight_invariance(name):
     y_pred = random_state.randint(0, 2, size=(n_samples,))
     y_score = random_state.random_sample(size=(n_samples,))
     metric = ALL_METRICS[name]
-    if name in CONTINUOUS_CLASSIFICATION_METRICS:
+    if name in (CONTINUOUS_CLASSIFICATION_METRICS | CURVE_METRICS.keys()):
         check_sample_weight_invariance(name, metric, y_true, y_score)
     else:
         check_sample_weight_invariance(name, metric, y_true, y_pred)
@@ -1816,7 +1833,7 @@ def test_no_averaging_labels():

 [USER_REDACTED].mark.parametrize(
-    "name", sorted(MULTILABELS_METRICS - {"unnormalized_multilabel_confusion_matrix"})
+    "name", sorted(MULTILABELS_METRICS - {"multilabel_confusion_matrix"})
 )
 def test_multilabel_label_permutations_invariance(name):
     random_state = check_random_state(0)

Thanks for the PR.

In `roc_curve` we do:

https://github.com/[USER_REDACTED]n/scikit-learn/blob/7f0900c265936eac9a89bba37eb19ee66208d46a/sklearn/metrics/_ranking.py#L1271-L1280

when scaling by a float, `diff` can result in extra indices. Note that in `confusion_matrix_at_thresholds` we already use max floating point precision when calculating `tps` and `fps` with `cumulative_sum`.

Also note `drop_intermediate=True` is the default.

<details open>
<summary>Details</summary>

In the test `test_binary_sample_weight_invariance`, with no scaling the indices are:

```
optimal_idxs=array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17,
18, 20, 21, 22, 23, 24, 25, 26, 29, 30, 32, 33, 34, 35, 36, 37, 38,
39, 41, 42, 43, 44, 45, 46, 48, 49])
```

with scaling 0.3, the indices are:

```
optimal_idxs=array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,
17, 18, 20, 21, 22, 23, 24, 25, 26, 29, 30, 32, 33, 34, 35, 36, 37,
38, 39, 40, 41, 42, 43, 44, 45, 46, 48, 49])
```

there are extra indices: 14 and 40 here

</details>

I think we should define a test module constant `WEIGHT_SCALE_DEPENDENT_METRICS` with all the metric names that are expected to be dependent on the weight scale.

Just found:

https://github.com/[USER_REDACTED]n/scikit-learn/blob/7f0900c265936eac9a89bba37eb19ee66208d46a/sklearn/metrics/tests/test_common.py#L159-L163

it seems the current procedure is to add "unnormalized" to the start of any non-partial metrics to indicate scale invariance.

I *think* it still would be nice to have `WEIGHT_SCALE_DEPENDENT_METRICS`, so you can see all the scale variant metrics in one place, but I am not 100% we shouldn't just stick to the old procedure...

Edit: Added, but can easily undo that commit, let me know what you think


FIX add quickfix for pandas warning causing CI error

<!--
🙌 Thanks for contributing a pull request!

👀 Please ensure you have taken a look at the contribution guidelines:
https://github.com/[USER_REDACTED]n/scikit-learn/blob/main/CONTRIBUTING.md

✅ In particular following the pull request checklist will increase the likelihood
of having maintainers review your PR:
https://scikit-learn.org/dev/developers/contributing.html#pull-request-checklist

📋 If your PR is likely to affect users, you will need to add a changelog entry
describing your PR changes, see:
https://github.com/[USER_REDACTED]n/scikit-learn/blob/main/doc/whats_new/upcoming_changes/README.md
-->

#### [NAME_REDACTED]/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes #32829

#### What does this implement/fix? Explain your changes.
Catches the `Pandas4Warning` that causes a CI failure.

#### Any other comments?
I'm not sure if this makes sense, but AFAIU the main thing the test is checking is whether `TargetEncoder` works when `y` is read-only, so maybe the test could even be changed into explicitly creating a read-only `y` without the implicit pandas conversion?

Edit: I changed the version check to use `sklearn.externals._packaging.version`.

[USER_REDACTED]

<!--
Thank you for your patience. Changes to scikit-learn require careful
attention, but with limited maintainer time, not every contribution can be reviewed
quickly.
For more information and tips on improving your pull request, see:
https://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

@@ -1,4 +1,5 @@
 import re
+import warnings

 import numpy as np
 import pytest
@@ -20,6 +21,7 @@
     LabelEncoder,
     TargetEncoder,
 )
+from sklearn.utils.fixes import parse_version

 def _encode_target(X_ordinal, y_numeric, n_categories, smooth):
@@ -709,6 +711,25 @@ def test_pandas_copy_on_write():
     Non-regression test for gh-27879.
     """
     pd = pytest.importorskip("pandas", minversion="2.0")
-    with pd.option_context("mode.copy_on_write", True):
+    # Pandas currently warns that setting copy_on_write will be removed in pandas 4
+    # (and copy-on-write will always be enabled).
+    # see https://github.com/[USER_REDACTED]n/scikit-learn/issues/32829
+    # TODO: remove this workaround when pandas 4 is our minimum version
+    if parse_version(pd.__version__) >= parse_version("4.0"):
         df = pd.DataFrame({"x": ["a", "b", "b"], "y": [4.0, 5.0, 6.0]})
         TargetEncoder(target_type="continuous").fit(df[["x"]], df["y"])
+    else:
+        with warnings.catch_warnings():
+            expected_message = (
+                "Copy-on-Write can no longer be disabled, "
+                "setting to False has no impact. This option will "
+                "be removed in pandas 4.0."
+            )
+            warnings.filterwarnings(
+                "ignore",
+                message=re.escape(expected_message),
+                category=DeprecationWarning,
+            )
+            with pd.option_context("mode.copy_on_write", True):
+                df = pd.DataFrame({"x": ["a", "b", "b"], "y": [4.0, 5.0, 6.0]})
+                TargetEncoder(target_type="continuous").fit(df[["x"]], df["y"])

Thanks for the PR!

I took a look at the class hierarchy of the warning in the nightly build of pandas, and it is a subclass of the usual DeprecationWarning:

```python
>>> pd.errors.Pandas4Warning.__mro__
(pandas.errors.Pandas4Warning,
 pandas.errors.PandasDeprecationWarning,
 pandas.errors.PandasChangeWarning,
 DeprecationWarning,
 Warning,
 Exception,
 BaseException,
 object)
```

So filtering those might work, irrespective of the Pandas version.

```suggestion
                category=DeprecationWarning,
```

Also: the `message` argument of `filterwarnings` is regular expression. So it's likely necessary to wrap the verbatim expected message as `re.escape(expected_warning_message)`.