# Make more of the "tools" of scikit-learn Array API compatible

🚨 🚧 This issue requires a bit of patience and experience to contribute to 🚧 🚨

- Original issue introducing array API in scikit-learn: #22352
- array API official doc/spec: https://data-apis.org/array-api/
- scikit-learn doc: https://scikit-learn.org/dev/modules/array_api.html

Please mention this issue when you create a PR, but please don't write "closes #26024" or "fixes #26024".

scikit-learn contains lots of useful tools, in addition to the many estimators it has. For example [metrics](https://scikit-learn.org/stable/modules/classes.html#sklearn-metrics-metrics), [pipelines](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.pipeline), [pre-processing](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.preprocessing) and [mode selection](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection). These are useful to and used by people who do not necessarily use an estimator from scikit-learn. This is great.

The fact that many users install scikit-learn "just" to use `train_test_split` is a testament to how useful it is to provide easy to use tools that do the right(!) thing. Instead of everyone implementing them from scratch because it is "easy" and making mistakes along the way.

In this issue I'd like to collect and track work related to making it easier to use all these "tools" from scikit-learn even if you are not using Numpy arrays for your data. In particular thanks to the Array API standard it should be "not too much work" to make things usable with data that is in an array that conforms to the Array API standard.

There is work in #25956 and #22554 which adds the basic infrastructure needed to use "array API arrays".

The goal of this issue is to make code like the following work:
```python
>>> from sklearn.preprocessing import MinMaxScaler
>>> from sklearn import config_context
>>> from sklearn.datasets import make_classification
>>> import torch
>>> X_np, y_np = make_classification(random_state=0)
>>> X_torch = torch.asarray(X_np, device="cuda", dtype=torch.float32)
>>> y_torch = torch.asarray(y_np, device="cuda", dtype=torch.float32)

>>> with config_context(array_api_dispatch=True):
...     # For example using MinMaxScaler on PyTorch tensors
...     scale = MinMaxScaler()
...     X_trans = scale.fit_transform(X_torch, y_torch)
...     assert type(X_trans) == type(X_torch)
...     assert X_trans.device == X_torch.device
```

The first step (or maybe part of the first) is to check which of them already "just work". After that is done we can start the work (one PR per class/function) making changes.

## Guidelines for testing

General comment: most of the time when we add array API support to a function in scikit-learn, we do not touch the existing (numpy-only) tests to make sure that the PR does not change the default behavior of scikit-learn on traditional inputs when array API is not enabled.

In the case of an estimator, it can be enough to add the `array_api_support=True` estimator tag in a method named `__sklearn_tags__`. For metric functions, just register it in the `array_api_metric_checkers` in `sklearn/metrics/tests/test_common.py` to include it in the common test.

For other kinds of functions not covered by existing common tests, or when the array API support depends heavily on non-default values, it might be required to add one or more new test functions to the related module-level test file. The general testing scheme is the following:

- generate some random test data with numpy or `sklearn.datasets.make_*`;
- call the function once on the numpy inputs without enabling array API dispatch;
- convert the inputs to a namespace / device combo passed as parameter to the test;
- call the function with array API dispatching enabled (under a `with sklearn.config_context(array_api_dispatch=True)` block
- check that the results are on the same namespace and device as the input
- convert back the output to a numpy array using `_convert_to_numpy`
- compare the original / reference numpy results and the `xp` computation results converted back to numpy using `assert_allclose` or similar.

Those tests should have `array_api` somewhere in their name to makes sure that we can run all the array API compliance tests with a keyword search in the pytest command line, e.g.:

```
pytest -k array_api sklearn/some/subpackage
```

In particular, for cost reasons, our CUDA GPU CI only runs `pytest -k array_api sklearn`. So it's very important to respect this naming conventions, otherwise we will not tests all what we are supposed to test on CUDA.

More generally, look at merged array API pull requests to see how testing is typically handled.

diff --git a/sklearn/metrics/tests/test_common.py b/sklearn/metrics/tests/test_common.py
index 353dfebf93bcf..0b7d8b474cec1 100644
--- a/sklearn/metrics/tests/test_common.py
+++ b/sklearn/metrics/tests/test_common.py
@@ -2098,6 +2098,18 @@ def check_array_api_multiclass_classification_metric(
     y_true_np = np.array([0, 1, 2, 3])
     y_pred_np = np.array([0, 1, 0, 2])

+    if metric.__name__ == "average_precision_score":
+        # we need y_pred_nd to be of shape (n_samples, n_classes)
+        y_pred_np = np.array(
+            [
+                [0.7, 0.2, 0.05, 0.05],
+                [0.1, 0.8, 0.05, 0.05],
+                [0.1, 0.1, 0.7, 0.1],
+                [0.05, 0.05, 0.1, 0.8],
+            ],
+            dtype=dtype_name,
+        )
+
     additional_params = {
         "average": ("micro", "macro", "weighted"),
         "beta": (0.2, 0.5, 0.8),
@@ -2299,6 +2311,11 @@ def check_array_api_metric_pairwise(metric, array_namespace, device, dtype_name)
         check_array_api_multiclass_classification_metric,
         check_array_api_multilabel_classification_metric,
     ],
+    average_precision_score: [
+        check_array_api_binary_classification_metric,
+        check_array_api_multiclass_classification_metric,
+        check_array_api_multilabel_classification_metric,
+    ],
     balanced_accuracy_score: [
         check_array_api_binary_classification_metric,
         check_array_api_multiclass_classification_metric,

diff --git a/sklearn/metrics/_base.py b/sklearn/metrics/_base.py
index c7668bce9fceb..9964929a446b5 100644
--- a/sklearn/metrics/_base.py
+++ b/sklearn/metrics/_base.py
@@ -10,7 +10,9 @@

 import numpy as np

+import sklearn.externals.array_api_extra as xpx
 from sklearn.utils import check_array, check_consistent_length
+from sklearn.utils._array_api import _average, _ravel, get_namespace_and_device
 from sklearn.utils.multiclass import type_of_target

@@ -19,6 +21,9 @@ def _average_binary_score(binary_metric, y_true, y_score, average, sample_weight

     Parameters
     ----------
+    binary_metric : callable, returns shape [n_classes]
+        The binary metric function to use.
+
     y_true : array, shape = [n_samples] or [n_samples, n_classes]
         True binary labels in binary label indicators.

@@ -47,9 +52,6 @@ def _average_binary_score(binary_metric, y_true, y_score, average, sample_weight
     sample_weight : array-like of shape (n_samples,), default=None
         Sample weights.

-    binary_metric : callable, returns shape [n_classes]
-        The binary metric function to use.
-
     Returns
     -------
     score : float or array of shape [n_classes]
@@ -57,6 +59,7 @@ def _average_binary_score(binary_metric, y_true, y_score, average, sample_weight
         classes.

     """
+    xp, _, _device = get_namespace_and_device(y_true, y_score, sample_weight)
     average_options = (None, "micro", "macro", "weighted", "samples")
     if average not in average_options:
         raise ValueError("average has to be one of {0}".format(average_options))
@@ -78,18 +81,23 @@ def _average_binary_score(binary_metric, y_true, y_score, average, sample_weight

     if average == "micro":
         if score_weight is not None:
-            score_weight = np.repeat(score_weight, y_true.shape[1])
-        y_true = y_true.ravel()
-        y_score = y_score.ravel()
+            score_weight = xp.repeat(score_weight, y_true.shape[1])
+        y_true = _ravel(y_true)
+        y_score = _ravel(y_score)

     elif average == "weighted":
         if score_weight is not None:
-            average_weight = np.sum(
-                np.multiply(y_true, np.reshape(score_weight, (-1, 1))), axis=0
+            #  Mixed integer and float type promotion not defined in array standard
+            y_true = xp.asarray(y_true, dtype=score_weight.dtype)
+            average_weight = xp.sum(
+                xp.multiply(y_true, xp.reshape(score_weight, (-1, 1))), axis=0
             )
         else:
-            average_weight = np.sum(y_true, axis=0)
-        if np.isclose(average_weight.sum(), 0.0):
+            average_weight = xp.sum(y_true, axis=0)
+        if xpx.isclose(
+            xp.sum(average_weight),
+            xp.asarray(0, dtype=average_weight.dtype, device=_device),
+        ):
             return 0

     elif average == "samples":
@@ -99,16 +107,20 @@ def _average_binary_score(binary_metric, y_true, y_score, average, sample_weight
         not_average_axis = 0

     if y_true.ndim == 1:
-        y_true = y_true.reshape((-1, 1))
+        y_true = xp.reshape(y_true, (-1, 1))

     if y_score.ndim == 1:
-        y_score = y_score.reshape((-1, 1))
+        y_score = xp.reshape(y_score, (-1, 1))

     n_classes = y_score.shape[not_average_axis]
-    score = np.zeros((n_classes,))
+    score = xp.zeros((n_classes,), device=_device)
     for c in range(n_classes):
-        y_true_c = y_true.take([c], axis=not_average_axis).ravel()
-        y_score_c = y_score.take([c], axis=not_average_axis).ravel()
+        y_true_c = _ravel(
+            xp.take(y_true, xp.asarray([c], device=_device), axis=not_average_axis)
+        )
+        y_score_c = _ravel(
+            xp.take(y_score, xp.asarray([c], device=_device), axis=not_average_axis)
+        )
         score[c] = binary_metric(y_true_c, y_score_c, sample_weight=score_weight)

     # Average the results
@@ -116,9 +128,8 @@ def _average_binary_score(binary_metric, y_true, y_score, average, sample_weight
         if average_weight is not None:
             # Scores with 0 weights are forced to be 0, preventing the average
             # score from being affected by 0-weighted NaN elements.
-            average_weight = np.asarray(average_weight)
             score[average_weight == 0] = 0
-        return float(np.average(score, weights=average_weight))
+        return float(_average(score, weights=average_weight, xp=xp))
     else:
         return score

diff --git a/sklearn/metrics/_ranking.py b/sklearn/metrics/_ranking.py
index 782f5c0fc7dbe..8712c63f0780a 100644
--- a/sklearn/metrics/_ranking.py
+++ b/sklearn/metrics/_ranking.py
@@ -31,6 +31,7 @@
 from sklearn.utils._array_api import (
     _max_precision_float_dtype,
     get_namespace_and_device,
+    move_to,
     size,
 )
 from sklearn.utils._encode import _encode, _unique
@@ -225,25 +226,36 @@ def average_precision_score(
     >>> average_precision_score(y_true, y_scores)
     0.77
     """
+    xp, _, device = get_namespace_and_device(y_score)
+    y_true, sample_weight = move_to(y_true, sample_weight, xp=xp, device=device)
+
+    if sample_weight is not None:
+        sample_weight = column_or_1d(sample_weight)

     def _binary_uninterpolated_average_precision(
-        y_true, y_score, pos_label=1, sample_weight=None
+        y_true,
+        y_score,
+        pos_label=1,
+        sample_weight=None,
+        xp=xp,
     ):
         precision, recall, _ = precision_recall_curve(
-            y_true, y_score, pos_label=pos_label, sample_weight=sample_weight
+            y_true,
+            y_score,
+            pos_label=pos_label,
+            sample_weight=sample_weight,
         )
         # Return the step function integral
         # The following works because the last entry of precision is
         # guaranteed to be 1, as returned by precision_recall_curve.
         # Due to numerical error, we can get `-0.0` and we therefore clip it.
-        return float(max(0.0, -np.sum(np.diff(recall) * np.array(precision)[:-1])))
+        return float(max(0.0, -xp.sum(xp.diff(recall) * precision[:-1])))

     y_type = type_of_target(y_true, input_name="y_true")
-
-    present_labels = np.unique(y_true)
+    present_labels = xp.unique_values(y_true)

     if y_type == "binary":
-        if len(present_labels) == 2 and pos_label not in present_labels:
+        if present_labels.shape[0] == 2 and pos_label not in present_labels:
             raise ValueError(
                 f"pos_label={pos_label} is not a valid label. It should be "
                 f"one of {present_labels}"
@@ -270,7 +282,7 @@ def _binary_uninterpolated_average_precision(
             )

     average_precision = partial(
-        _binary_uninterpolated_average_precision, pos_label=pos_label
+        _binary_uninterpolated_average_precision, pos_label=pos_label, xp=xp
     )
     return _average_binary_score(
         average_precision, y_true, y_score, average, sample_weight=sample_weight
@@ -686,6 +698,8 @@ class scores must correspond to the order of ``labels``,
     y_type = type_of_target(y_true, input_name="y_true")
     y_true = check_array(y_true, ensure_2d=False, dtype=None)
     y_score = check_array(y_score, ensure_2d=False)
+    if sample_weight is not None:
+        sample_weight = column_or_1d(sample_weight)

     if y_type == "multiclass" or (
         y_type == "binary" and y_score.ndim == 2 and y_score.shape[1] > 2
@@ -1142,7 +1156,7 @@ def precision_recall_curve(
             "No positive class found in y_true, "
             "recall is set to one for all thresholds."
         )
-        recall = xp.full(tps.shape, 1.0)
+        recall = xp.full(tps.shape, 1.0, device=device)
     else:
         recall = tps / tps[-1]


# Gradient Boosting: Tree splitting criterion`"friedman_mse"` isn't different from normal `"mse"`

### Describe the bug

In `sklearn/tree/_criterion.pyx`, criteria `"friedman_mse"` (class `FriedmanMSE(MSE)`) and `"squared_error"` (class `MSE`) are mathematically equivalent, they use different but equivalent formulas. This is confusing, and I think that:
- `"friedman_mse"` criterion should be deprecated then removed.
- the class should be removed

Because `"friedman_mse"` and `"squared_error"` are mathematically equivalent they produce the same trees/forests (up to float precision errors), as shown by the reproducer below.

Also, I tried removing all the code from `FriedmanMSE` class (leaving only the inheritance `FriedmanMSE(MSE)`) and all test passes from `sklearn/tree` and `sklearn/ensemble` except `test_huber_exact_backward_compat` which is sensitive to float rounding errors.
I also tried the other-way around (copy-pasting code from `FriedmanMSE(MSE)` in `MSE`) and except for one bug for which I'll open a separate issue, it worked as well.

Maths (from ChatGPT, but I carefully double-checked it):

Proof:

### Steps/Code to Reproduce

```Python
import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor

params_list = [
    {
        'random_state': 0,
        'max_leaf_nodes': max_leaf_nodes,
        'max_depth': max_depth,
        'ccp_alpha': ccp_alpha,
    }
    for max_depth in [None, 1, 2, 3, 5, 10]
    for max_leaf_nodes in [None, 3, 6, 10, 20, 50, 100]
    for ccp_alpha in [0, 0.005, 0.01]
]

for Reg in [DecisionTreeRegressor, RandomForestRegressor, GradientBoostingRegressor]:
    print(Reg)
    for params in params_list:
        n = np.random.choice([10, 100, 1000, 10_000])
        d = np.random.choice([1, 2, 3, 10])
        if Reg in [GradientBoostingRegressor, RandomForestRegressor]:
            params = {**params, 'n_estimators': 10}
        X = np.random.rand(n, 1)
        X_test = np.random.rand(n, 1)
        y = np.random.rand(n) + X.sum(axis=1)

        for w  in [None, np.random.rand(n), np.random.pareto(1, size=n)]:
            tree_mse = Reg(criterion="squared_error", **params)
            tree_friedman = Reg(criterion="friedman_mse",  **params)

            assert np.allclose(
                tree_mse.fit(X, y, sample_weight=w).predict(X_test),
                tree_friedman.fit(X, y, sample_weight=w).predict(X_test)
            )
            if Reg is DecisionTreeRegressor:
                assert np.allclose(tree_mse.tree_.impurity, tree_friedman.tree_.impurity)
```

### Expected Results

You would expect different criteria to give different results and hence this code to raise.

### Actual Results

Doesn't raise.

### Versions

```shell
System:
    python: 3.12.11 (main, Aug 18 2025, 19:19:11) [Clang 20.1.4 ]
executable: /home/arthur/dev-perso/scikit-learn/sklearn-env/bin/python
   machine: Linux-6.14.0-35-generic-x86_64-with-glibc2.39

Python dependencies:
      sklearn: 1.8.dev0
          pip: None
   setuptools: 80.9.0
        numpy: 2.3.4
        scipy: 1.16.2
       Cython: 3.1.5
       pandas: 2.3.3
   matplotlib: 3.10.7
       joblib: 1.5.2
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libscipy_openblas
       filepath: /home/arthur/dev-perso/scikit-learn/sklearn-env/lib/python3.12/site-packages/numpy.libs/libscipy_openblas64_-8fb3d286.so
        version: 0.3.30
threading_layer: pthreads
   architecture: Haswell

       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libscipy_openblas
       filepath: /home/arthur/dev-perso/scikit-learn/sklearn-env/lib/python3.12/site-packages/scipy.libs/libscipy_openblas-b75cc656.so
        version: 0.3.29.dev
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 16
         prefix: libgomp
       filepath: /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0
        version: None
```

diff --git a/sklearn/ensemble/tests/test_forest.py b/sklearn/ensemble/tests/test_forest.py
index 20a452ecb75c4..b04c46c845d27 100644
--- a/sklearn/ensemble/tests/test_forest.py
+++ b/sklearn/ensemble/tests/test_forest.py
@@ -157,8 +157,12 @@ def test_iris_criterion(name, criterion):
     assert score > 0.5, "Failed with criterion %s and score = %f" % (criterion, score)

+# TODO(1.11): remove the deprecated friedman_mse criterion parametrization
+@pytest.mark.filterwarnings("ignore:.*friedman_mse.*:FutureWarning")
 @pytest.mark.parametrize("name", FOREST_REGRESSORS)
-@pytest.mark.parametrize("criterion", ("squared_error", "absolute_error"))
+@pytest.mark.parametrize(
+    "criterion", ("squared_error", "friedman_mse", "absolute_error")
+)
 def test_regression_criterion(name, criterion):
     # Check consistency on regression dataset.
     ForestRegressor = FOREST_REGRESSORS[name]
@@ -1864,3 +1868,10 @@ def test_non_supported_criterion_raises_error_with_missing_values(Forest):
     msg = ".*does not accept missing values"
     with pytest.raises(ValueError, match=msg):
         forest.fit(X, y)
+
+
+# TODO(1.11): remove test with the deprecation of friedman_mse criterion
+@pytest.mark.parametrize("Forest", FOREST_REGRESSORS.values())
+def test_friedman_mse_deprecation(Forest):
+    with pytest.warns(FutureWarning, match="friedman_mse"):
+        _ = Forest(criterion="friedman_mse")

diff --git a/sklearn/ensemble/_forest.py b/sklearn/ensemble/_forest.py
index b7b4707dcaa0e..e1d666a8bf50f 100644
--- a/sklearn/ensemble/_forest.py
+++ b/sklearn/ensemble/_forest.py
@@ -1928,6 +1928,16 @@ def __init__(
             max_samples=max_samples,
         )

+        if isinstance(criterion, str) and criterion == "friedman_mse":
+            # TODO(1.11): remove support of "friedman_mse" criterion.
+            criterion = "squared_error"
+            warn(
+                'Value `"friedman_mse"` for `criterion` is deprecated and will be '
+                'removed in 1.11. It maps to `"squared_error"` as both '
+                'were always equivalent. Use `criterion="squared_error"` '
+                "to remove this warning.",
+                FutureWarning,
+            )
         self.criterion = criterion
         self.max_depth = max_depth
         self.min_samples_split = min_samples_split
@@ -2662,6 +2672,16 @@ def __init__(
             max_samples=max_samples,
         )

+        if isinstance(criterion, str) and criterion == "friedman_mse":
+            # TODO(1.11): remove support of "friedman_mse" criterion.
+            criterion = "squared_error"
+            warn(
+                'Value `"friedman_mse"` for `criterion` is deprecated and will be '
+                'removed in 1.11. It maps to `"squared_error"` as both '
+                'were always equivalent. Use `criterion="squared_error"` '
+                "to remove this warning.",
+                FutureWarning,
+            )
         self.criterion = criterion
         self.max_depth = max_depth
         self.min_samples_split = min_samples_split


# Fix `_safe_indexing` with non integer arrays on array API inputs

This is a fix for #32837.

While investigating the issue above, I realized that we needed unittest for array API support for `_safe_indexing`.

I am not yet sure if this problem already existing in 1.7 or not. If it was I will add a changelog entry.

diff --git a/sklearn/utils/tests/test_array_api.py b/sklearn/utils/tests/test_array_api.py
index a1fc81c109af8..785bb668e9878 100644
--- a/sklearn/utils/tests/test_array_api.py
+++ b/sklearn/utils/tests/test_array_api.py
@@ -50,8 +50,8 @@
 from sklearn.utils.fixes import _IS_32BIT, CSR_CONTAINERS, np_version, parse_version

-@pytest.mark.parametrize("X", [numpy.asarray([1, 2, 3]), [1, 2, 3]])
-def test_get_namespace_ndarray_default(X):
+@pytest.mark.parametrize("X", [numpy.asarray([1, 2, 3]), [1, 2, 3], (1, 2, 3)])
+def test_get_namespace_ndarray_or_similar_default(X):
     """Check that get_namespace returns NumPy wrapper"""
     xp_out, is_array_api_compliant = get_namespace(X)
     assert xp_out is np_compat
@@ -71,14 +71,13 @@ def test_get_namespace_ndarray_creation_device():

 @skip_if_array_api_compat_not_configured
-def test_get_namespace_ndarray_with_dispatch():
+@pytest.mark.parametrize("X", [numpy.asarray([1, 2, 3]), [1, 2, 3], (1, 2, 3)])
+def test_get_namespace_ndarray_or_similar_default_with_dispatch(X):
     """Test get_namespace on NumPy ndarrays."""

-    X_np = numpy.asarray([[1, 2, 3]])
-
     with config_context(array_api_dispatch=True):
-        xp_out, is_array_api_compliant = get_namespace(X_np)
-        assert is_array_api_compliant
+        xp_out, is_array_api_compliant = get_namespace(X)
+        assert is_array_api_compliant == isinstance(X, numpy.ndarray)

         # In the future, NumPy should become API compliant library and we should have
         # assert xp_out is numpy

diff --git a/sklearn/decomposition/_dict_learning.py b/sklearn/decomposition/_dict_learning.py
index d4550e4ce8982..2a32ad92de83e 100644
--- a/sklearn/decomposition/_dict_learning.py
+++ b/sklearn/decomposition/_dict_learning.py
@@ -1360,7 +1360,7 @@ def fit(self, X, y=None):
         if X.shape[1] != self.dictionary.shape[1]:
             raise ValueError(
                 "Dictionary and X have different numbers of features:"
-                f"dictionary.shape: {self.dictionary.shape} X.shape{X.shape}"
+                f"dictionary.shape: {self.dictionary.shape} X.shape: {X.shape}"
             )
         return self

diff --git a/sklearn/utils/_array_api.py b/sklearn/utils/_array_api.py
index 8bcf8bde132ea..23239ee062267 100644
--- a/sklearn/utils/_array_api.py
+++ b/sklearn/utils/_array_api.py
@@ -23,6 +23,11 @@
 __all__ = ["xpx"]  # we import xpx here just to re-export it, need this to appease ruff

 _NUMPY_NAMESPACE_NAMES = {"numpy", "sklearn.externals.array_api_compat.numpy"}
+REMOVE_TYPES_DEFAULT = (
+    str,
+    list,
+    tuple,
+)

 def yield_namespaces(include_numpy_namespaces=True):
@@ -167,7 +172,7 @@ def _single_array_device(array):
         return array.device

-def device(*array_list, remove_none=True, remove_types=(str,)):
+def device(*array_list, remove_none=True, remove_types=REMOVE_TYPES_DEFAULT):
     """Hardware device where the array data resides on.

     If the hardware device is not the same for all arrays, an error is raised.
@@ -180,7 +185,7 @@ def device(*array_list, remove_none=True, remove_types=(str,)):
     remove_none : bool, default=True
         Whether to ignore None objects passed in array_list.

-    remove_types : tuple or list, default=(str,)
+    remove_types : tuple or list, default=(str, list, tuple)
         Types to ignore in array_list.

     Returns
@@ -290,7 +295,7 @@ def supported_float_dtypes(xp, device=None):
     return tuple(valid_float_dtypes)

-def _remove_non_arrays(*arrays, remove_none=True, remove_types=(str,)):
+def _remove_non_arrays(*arrays, remove_none=True, remove_types=REMOVE_TYPES_DEFAULT):
     """Filter arrays to exclude None and/or specific types.

     Sparse arrays are always filtered out.
@@ -303,7 +308,7 @@ def _remove_non_arrays(*arrays, remove_none=True, remove_types=(str,)):
     remove_none : bool, default=True
         Whether to ignore None objects passed in arrays.

-    remove_types : tuple or list, default=(str,)
+    remove_types : tuple or list, default=(str, list, tuple)
         Types to ignore in the arrays.

     Returns
@@ -328,7 +333,27 @@ def _remove_non_arrays(*arrays, remove_none=True, remove_types=(str,)):
     return filtered_arrays

-def get_namespace(*arrays, remove_none=True, remove_types=(str,), xp=None):
+def _unwrap_memoryviewslices(*arrays):
+    # Since _cyutility._memoryviewslice is an implementation detail of the
+    # Cython runtime, we would rather not introduce a possibly brittle
+    # import statement to run `isinstance`-based filtering, hence the
+    # attribute-based type inspection.
+    unwrapped = []
+    for a in arrays:
+        a_type = type(a)
+        if (
+            a_type.__module__ == "_cyutility"
+            and a_type.__name__ == "_memoryviewslice"
+            and hasattr(a, "base")
+        ):
+            a = a.base
+        unwrapped.append(a)
+    return unwrapped
+
+
+def get_namespace(
+    *arrays, remove_none=True, remove_types=REMOVE_TYPES_DEFAULT, xp=None
+):
     """Get namespace of arrays.

     Introspect `arrays` arguments and return their common Array API compatible
@@ -364,7 +389,7 @@ def get_namespace(*arrays, remove_none=True, remove_types=(str,), xp=None):
     remove_none : bool, default=True
         Whether to ignore None objects passed in arrays.

-    remove_types : tuple or list, default=(str,)
+    remove_types : tuple or list, default=(str, list, tuple)
         Types to ignore in the arrays.

     xp : module, default=None
@@ -399,12 +424,19 @@ def get_namespace(*arrays, remove_none=True, remove_types=(str,), xp=None):
         remove_types=remove_types,
     )

+    # get_namespace can be called by helper functions that are used both in
+    # array API compatible code and non-array API Cython related code. To
+    # support the latter on NumPy inputs without raising a TypeError, we
+    # unwrap potential Cython memoryview slices here.
+    arrays = _unwrap_memoryviewslices(*arrays)
+
     if not arrays:
         return np_compat, False

     _check_array_api_dispatch(array_api_dispatch)

-    namespace, is_array_api_compliant = array_api_compat.get_namespace(*arrays), True
+    namespace = array_api_compat.get_namespace(*arrays)
+    is_array_api_compliant = True

     if namespace.__name__ == "array_api_strict" and hasattr(
         namespace, "set_array_api_strict_flags"
@@ -415,7 +447,7 @@ def get_namespace(*arrays, remove_none=True, remove_types=(str,), xp=None):

 def get_namespace_and_device(
-    *array_list, remove_none=True, remove_types=(str,), xp=None
+    *array_list, remove_none=True, remove_types=REMOVE_TYPES_DEFAULT, xp=None
 ):
     """Combination into one single function of `get_namespace` and `device`.

@@ -425,7 +457,7 @@ def get_namespace_and_device(
         Array objects.
     remove_none : bool, default=True
         Whether to ignore None objects passed in arrays.
-    remove_types : tuple or list, default=(str,)
+    remove_types : tuple or list, default=(str, list, tuple)
         Types to ignore in the arrays.
     xp : module, default=None
         Precomputed array namespace module. When passed, typically from a caller
diff --git a/sklearn/utils/_test_common/instance_generator.py b/sklearn/utils/_test_common/instance_generator.py
index 14f8090b96cf8..176d1ab070ca6 100644
--- a/sklearn/utils/_test_common/instance_generator.py
+++ b/sklearn/utils/_test_common/instance_generator.py
@@ -46,7 +46,10 @@
     SparsePCA,
     TruncatedSVD,
 )
-from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
+from sklearn.discriminant_analysis import (
+    LinearDiscriminantAnalysis,
+    QuadraticDiscriminantAnalysis,
+)
 from sklearn.dummy import DummyClassifier
 from sklearn.ensemble import (
     AdaBoostClassifier,
@@ -79,6 +82,7 @@
     SequentialFeatureSelector,
 )
 from sklearn.frozen import FrozenEstimator
+from sklearn.impute import SimpleImputer
 from sklearn.kernel_approximation import (
     Nystroem,
     PolynomialCountSketch,
@@ -559,11 +563,16 @@
             dict(solver="lbfgs"),
         ],
     },
-    GaussianMixture: {"check_dict_unchanged": dict(max_iter=5, n_init=2)},
+    GaussianMixture: {
+        "check_dict_unchanged": dict(max_iter=5, n_init=2),
+        "check_array_api_input": dict(
+            max_iter=5, n_init=2, init_params="random_from_data"
+        ),
+    },
     GaussianRandomProjection: {"check_dict_unchanged": dict(n_components=1)},
+    GraphicalLasso: {"check_array_api_input": dict(max_iter=5, alpha=1.0)},
     IncrementalPCA: {"check_dict_unchanged": dict(batch_size=10, n_components=1)},
     Isomap: {"check_dict_unchanged": dict(n_components=1)},
-    KMeans: {"check_dict_unchanged": dict(max_iter=5, n_clusters=1, n_init=2)},
     # TODO(1.9) simplify when averaged_inverted_cdf is the default
     KBinsDiscretizer: {
         "check_sample_weight_equivalence_on_dense_data": [
@@ -595,7 +604,11 @@
             strategy="quantile", quantile_method="averaged_inverted_cdf"
         ),
     },
-    KernelPCA: {"check_dict_unchanged": dict(n_components=1)},
+    KernelPCA: {
+        "check_dict_unchanged": dict(n_components=1),
+        "check_array_api_input": dict(fit_inverse_transform=True),
+    },
+    KMeans: {"check_dict_unchanged": dict(max_iter=5, n_clusters=1, n_init=2)},
     LassoLars: {"check_non_transformer_estimators_n_iter": dict(alpha=0.0)},
     LatentDirichletAllocation: {
         "check_dict_unchanged": dict(batch_size=10, max_iter=5, n_components=1)
@@ -693,6 +706,7 @@
             dict(solver="highs-ipm"),
         ],
     },
+    QuadraticDiscriminantAnalysis: {"check_array_api_input": dict(reg_param=1.0)},
     RBFSampler: {"check_dict_unchanged": dict(n_components=1)},
     Ridge: {
         "check_sample_weight_equivalence_on_dense_data": [
@@ -720,7 +734,9 @@
         ],
     },
     SkewedChi2Sampler: {"check_dict_unchanged": dict(n_components=1)},
+    SimpleImputer: {"check_array_api_input": dict(add_indicator=True)},
     SparseCoder: {
+        "check_array_api_input": dict(dictionary=rng.normal(size=(5, 10))),
         "check_estimators_dtypes": dict(dictionary=rng.normal(size=(5, 5))),
         "check_dtype_object": dict(dictionary=rng.normal(size=(5, 10))),
         "check_transformers_unfitted_stateless": dict(
diff --git a/sklearn/utils/estimator_checks.py b/sklearn/utils/estimator_checks.py
index 7fd36041f608a..3d166d875fb6c 100644
--- a/sklearn/utils/estimator_checks.py
+++ b/sklearn/utils/estimator_checks.py
@@ -196,9 +196,11 @@ def _yield_checks(estimator):
     yield check_estimators_pickle
     yield partial(check_estimators_pickle, readonly_memmap=True)

-    if tags.array_api_support:
-        for check in _yield_array_api_checks(estimator):
-            yield check
+    for check in _yield_array_api_checks(
+        estimator,
+        only_numpy=not tags.array_api_support,
+    ):
+        yield check

     yield check_f_contiguous_array_estimator

@@ -336,18 +338,30 @@ def _yield_outliers_checks(estimator):
     yield check_non_transformer_estimators_n_iter

-def _yield_array_api_checks(estimator):
-    for (
-        array_namespace,
-        device,
-        dtype_name,
-    ) in yield_namespace_device_dtype_combinations():
+def _yield_array_api_checks(estimator, only_numpy=False):
+    if only_numpy:
+        # Enabling array API dispatch and using NumPy inputs should not
+        # change results, even if the estimator does not explicitly support
+        # array API.
         yield partial(
             check_array_api_input,
-            array_namespace=array_namespace,
-            dtype_name=dtype_name,
-            device=device,
+            array_namespace="numpy",
+            expect_only_array_outputs=False,
         )
+    else:
+        # These extended checks should pass for all estimators that declare
+        # array API support in their tags.
+        for (
+            array_namespace,
+            device,
+            dtype_name,
+        ) in yield_namespace_device_dtype_combinations():
+            yield partial(
+                check_array_api_input,
+                array_namespace=array_namespace,
+                dtype_name=dtype_name,
+                device=device,
+            )

 def _yield_all_checks(estimator, legacy: bool):
@@ -1048,6 +1062,7 @@ def check_array_api_input(
     dtype_name="float64",
     check_values=False,
     check_sample_weight=False,
+    expect_only_array_outputs=True,
 ):
     """Check that the estimator can work consistently with the Array API

@@ -1057,17 +1072,25 @@ def check_array_api_input(
     When check_values is True, it also checks that calling the estimator on the
     array_api Array gives the same results as ndarrays.

-    When sample_weight is True, dummy sample weights are passed to the fit call.
+    When check_sample_weight is True, dummy sample weights are passed to the
+    fit call.
+
+    When expect_only_array_outputs is False, the check is looser: in particular
+    it accepts non-array outputs such as sparse data structures. This is
+    useful to test that enabling array API dispatch does not change the
+    behavior of any estimator fed with NumPy inputs, even for estimators that
+    do not support array API.
     """
     xp = _array_api_for_tests(array_namespace, device)

-    X, y = make_classification(random_state=42)
+    X, y = make_classification(n_samples=30, n_features=10, random_state=42)
     X = X.astype(dtype_name, copy=False)

     X = _enforce_estimator_tags_X(estimator_orig, X)
     y = _enforce_estimator_tags_y(estimator_orig, y)

     est = clone(estimator_orig)
+    set_random_state(est)

     X_xp = xp.asarray(X, device=device)
     y_xp = xp.asarray(y, device=device)
@@ -1193,47 +1216,48 @@ def check_array_api_input(
             f"got {result_ns}."
         )

-        with config_context(array_api_dispatch=True):
-            assert array_device(result_xp) == array_device(X_xp)
-
-        result_xp_np = _convert_to_numpy(result_xp, xp=xp)
+        if expect_only_array_outputs:
+            with config_context(array_api_dispatch=True):
+                assert array_device(result_xp) == array_device(X_xp)

-        if check_values:
-            assert_allclose(
-                result,
-                result_xp_np,
-                err_msg=f"{method} did not the return the same result",
-                atol=_atol_for_type(X.dtype),
-            )
-        else:
-            if hasattr(result, "shape"):
+            result_xp_np = _convert_to_numpy(result_xp, xp=xp)
+            if check_values:
+                assert_allclose(
+                    result,
+                    result_xp_np,
+                    err_msg=f"{method} did not the return the same result",
+                    atol=_atol_for_type(X.dtype),
+                )
+            elif hasattr(result, "shape"):
                 assert result.shape == result_xp_np.shape
                 assert result.dtype == result_xp_np.dtype

         if method_name == "transform" and hasattr(est, "inverse_transform"):
             inverse_result = est.inverse_transform(result)
             with config_context(array_api_dispatch=True):
-                invese_result_xp = est_xp.inverse_transform(result_xp)
-                inverse_result_ns = get_namespace(invese_result_xp)[0].__name__
-            assert inverse_result_ns == input_ns, (
-                "'inverse_transform' output is in wrong namespace, expected"
-                f" {input_ns}, got {inverse_result_ns}."
-            )
-
-            with config_context(array_api_dispatch=True):
-                assert array_device(invese_result_xp) == array_device(X_xp)
-
-            invese_result_xp_np = _convert_to_numpy(invese_result_xp, xp=xp)
-            if check_values:
-                assert_allclose(
-                    inverse_result,
-                    invese_result_xp_np,
-                    err_msg="inverse_transform did not the return the same result",
-                    atol=_atol_for_type(X.dtype),
+                inverse_result_xp = est_xp.inverse_transform(result_xp)
+
+            if expect_only_array_outputs:
+                with config_context(array_api_dispatch=True):
+                    inverse_result_ns = get_namespace(inverse_result_xp)[0].__name__
+                assert inverse_result_ns == input_ns, (
+                    "'inverse_transform' output is in wrong namespace, expected"
+                    f" {input_ns}, got {inverse_result_ns}."
                 )
-            else:
-                assert inverse_result.shape == invese_result_xp_np.shape
-                assert inverse_result.dtype == invese_result_xp_np.dtype
+                with config_context(array_api_dispatch=True):
+                    assert array_device(result_xp) == array_device(X_xp)
+
+                inverse_result_xp_np = _convert_to_numpy(inverse_result_xp, xp=xp)
+                if check_values:
+                    assert_allclose(
+                        inverse_result,
+                        inverse_result_xp_np,
+                        err_msg="inverse_transform did not the return the same result",
+                        atol=_atol_for_type(X.dtype),
+                    )
+                elif hasattr(result, "shape"):
+                    assert inverse_result.shape == inverse_result_xp_np.shape
+                    assert inverse_result.dtype == inverse_result_xp_np.dtype

 def check_array_api_input_and_values(


# FEA Add array API support for `average_precision_score`

#### Reference Issues/PRs
towards #26024

#### What does this implement/fix? Explain your changes.
Adds array API support to `average_precision_score`

#### AI usage disclosure
<!--
If AI tools were involved in creating this PR, please check all boxes that apply
below and make sure that you adhere to our Automated Contributions Policy:
https://scikit-learn.org/dev/developers/contributing.html#automated-contributions-policy
-->
I used AI assistance for:
- [ ] Code generation (e.g., when writing an implementation or fixing a bug)
- [ ] Test/benchmark generation
- [ ] Documentation (including examples)
- [x] Research and understanding

#### Any other comments?
WIP

TODO:
- [x] add array API in `average_precision_score`
- [x] add common metrics test
- [x] add average_precision_score to the array API docs
- [x] removed old fix for numpy<= 1.24.1 from `average_precision_score` as well as from `_check_set_wise_labels` which allows to have `present_labels` as an array instead of a list
- [x] use `LabelBinarizer` instead of `label_binarize` # TODO: check if in fact necessary
- [x] add array API in `_average_binary_score`
- [x] fix remaining test failures
- [x] add changelog
- [x] make it accept mixed namespace inputs
- [ ] (only necessary if #32755 is merged first) add new test case for `average_precision_score` in `METRICS_SUPPORTING_MIXED_NAMESPACE` in `sklearn/metrics/tests/test_common.py`

diff --git a/sklearn/metrics/tests/test_ranking.py b/sklearn/metrics/tests/test_ranking.py
index 821537571ea1a..fb607d319482f 100644
--- a/sklearn/metrics/tests/test_ranking.py
+++ b/sklearn/metrics/tests/test_ranking.py
@@ -1212,7 +1212,7 @@ def test_average_precision_score_multilabel_pos_label_errors():
 def test_average_precision_score_multiclass_pos_label_errors():
     # Raise an error for multiclass y_true with pos_label other than 1
     y_true = np.array([0, 1, 2, 0, 1, 2])
-    y_pred = np.array(
+    y_score = np.array(
         [
             [0.5, 0.2, 0.1],
             [0.4, 0.5, 0.3],
@@ -1227,7 +1227,21 @@ def test_average_precision_score_multiclass_pos_label_errors():
         "Do not set pos_label or set pos_label to 1."
     )
     with pytest.raises(ValueError, match=err_msg):
-        average_precision_score(y_true, y_pred, pos_label=3)
+        average_precision_score(y_true, y_score, pos_label=3)
+
+
+def test_multiclass_ranking_metrics_raise_for_incorrect_shape_of_y_score():
+    """Test ranking metrics, with multiclass support, raise if shape `y_score` is 1D."""
+    y_true = np.array([0, 1, 2, 0, 1, 2])
+    y_score = np.array([0.5, 0.4, 0.8, 0.9, 0.8, 0.7])
+
+    msg = re.escape("`y_score` needs to be of shape `(n_samples, n_classes)`")
+    with pytest.raises(ValueError, match=msg):
+        average_precision_score(y_true, y_score)
+    with pytest.raises(ValueError, match=msg):
+        roc_auc_score(y_true, y_score, multi_class="ovr")
+    with pytest.raises(ValueError, match=msg):
+        top_k_accuracy_score(y_true, y_score)

 def test_score_scale_invariance():

diff --git a/sklearn/metrics/_ranking.py b/sklearn/metrics/_ranking.py
index 8226e49bff6d0..782f5c0fc7dbe 100644
--- a/sklearn/metrics/_ranking.py
+++ b/sklearn/metrics/_ranking.py
@@ -142,7 +142,8 @@ def average_precision_score(
     Parameters
     ----------
     y_true : array-like of shape (n_samples,) or (n_samples, n_classes)
-        True binary labels or :term:`multilabel indicator matrix`.
+        True binary labels, :term:`multi-label` indicators (as a
+        :term:`multilabel indicator matrix`) or :term:`multi-class` labels.

     y_score : array-like of shape (n_samples,) or (n_samples, n_classes)
         Target scores, can either be probability estimates of the positive
@@ -261,6 +262,12 @@ def _binary_uninterpolated_average_precision(
                 "Do not set pos_label or set pos_label to 1."
             )
         y_true = label_binarize(y_true, classes=present_labels)
+        if not y_score.shape == y_true.shape:
+            raise ValueError(
+                "`y_score` needs to be of shape `(n_samples, n_classes)`, since "
+                "`y_true` contains multiple classes. Got "
+                f"`y_score.shape={y_score.shape}`."
+            )

     average_precision = partial(
         _binary_uninterpolated_average_precision, pos_label=pos_label
@@ -764,7 +771,12 @@ def _multiclass_roc_auc_score(
         Sample weights.

     """
-    # validation of the input y_score
+    if not y_score.ndim == 2:
+        raise ValueError(
+            "`y_score` needs to be of shape `(n_samples, n_classes)`, since "
+            "`y_true` contains multiple classes. Got "
+            f"`y_score.shape={y_score.shape}`."
+        )
     if not np.allclose(1, y_score.sum(axis=1)):
         raise ValueError(
             "Target scores need to be probabilities for multiclass "
@@ -2111,6 +2123,13 @@ def top_k_accuracy_score(
                 " labels, `labels` must be provided."
             )
         y_score = column_or_1d(y_score)
+    else:
+        if not y_score.ndim == 2:
+            raise ValueError(
+                "`y_score` needs to be of shape `(n_samples, n_classes)`, since "
+                "`y_true` contains multiple classes. Got "
+                f"`y_score.shape={y_score.shape}`."
+            )

     check_consistent_length(y_true, y_score, sample_weight)
     y_score_n_classes = y_score.shape[1] if y_score.ndim == 2 else 2


# BUG: fix pickling of SigmaClip when Bottleneck is installed

Fixes #19343

When Bottleneck is installed, `SigmaClip` could fail to pickle because `_dtype_dispatch` returned a local closure, which Python cannot pickle. Since these functions are stored on the `SigmaClip` instance (`_cenfunc_parsed` and `_stdfunc_parsed`), pickling the object raised an `AttributeError`.

Replaced the closure with a small `_DtypeDispatch` class that performs the same dtype-based dispatch but is picklable. A regression test was also added to ensure that `SigmaClip` objects can be pickled correctly.

diff --git a/astropy/stats/tests/test_sigma_clipping.py b/astropy/stats/tests/test_sigma_clipping.py
index 501273e07299..9e95bd77e2f6 100644
--- a/astropy/stats/tests/test_sigma_clipping.py
+++ b/astropy/stats/tests/test_sigma_clipping.py
@@ -1,5 +1,7 @@
 # Licensed under a 3-clause BSD style license - see LICENSE.rst

+import pickle
+
 import numpy as np
 import pytest
 from numpy.testing import assert_allclose, assert_equal
@@ -722,3 +724,18 @@ def test_propagation_of_mask():
     y = np.ma.masked_where(x > 1, x)

     assert_allclose(sigma_clipped_stats(y, grow=1), (1, 1, 0))
+
+
+def test_sigmaclip_pickle():
+    """Regression test for gh-19343: SigmaClip cannot be pickled
+    when Bottleneck is installed."""
+
+    sigclip = SigmaClip(sigma=3.0, maxiters=5)
+    restored = pickle.loads(pickle.dumps(sigclip))
+    assert restored.sigma == sigclip.sigma
+    assert restored.maxiters == sigclip.maxiters
+    # verify the restored object still works correctly
+    data = np.ma.array([1, 2, 3, 100, 4, 5])
+    result = restored(data)
+    expected = sigclip(data)
+    np.testing.assert_array_equal(result, expected)

diff --git a/astropy/stats/nanfunctions.py b/astropy/stats/nanfunctions.py
index efa9679d401b..20f64eb85def 100644
--- a/astropy/stats/nanfunctions.py
+++ b/astropy/stats/nanfunctions.py
@@ -101,29 +101,38 @@ def _apply_bottleneck(
         nanvar=np.nanvar,
     )

-    def _dtype_dispatch(func_name):
-        # dispatch to bottleneck or numpy depending on the input array dtype
-        # this is done to workaround known accuracy bugs in bottleneck
-        # affecting float32 calculations
-        # see https://github.com/pydata/bottleneck/issues/379
-        # see https://github.com/pydata/bottleneck/issues/462
-        # see https://github.com/astropy/astropy/issues/17185
-        # see https://github.com/astropy/astropy/issues/11492
-        def wrapped(*args, **kwargs):
-            if args[0].dtype.str[1:] == "f8":
-                return bn_funcs[func_name](*args, **kwargs)
-            else:
-                return np_funcs[func_name](*args, **kwargs)
+    class _DtypeDispatch:
+        """Picklable dispatcher that routes to bottleneck or numpy based on dtype.
+
+        Using a class instead of a closure ensures instances can be pickled,
+        which is required for pickling objects that store these functions as
+        attributes (e.g., SigmaClip._cenfunc_parsed).
+
+        Bottleneck is only used for float64 (f8) arrays to work around known
+        accuracy bugs affecting float32 calculations:
+        - https://github.com/pydata/bottleneck/issues/379
+        - https://github.com/pydata/bottleneck/issues/462
+        - https://github.com/astropy/astropy/issues/17185
+        - https://github.com/astropy/astropy/issues/11492
+        """

-        return wrapped
+        def __init__(self, func_name):
+            self.func_name = func_name

-    nansum = _dtype_dispatch("nansum")
-    nanmin = _dtype_dispatch("nanmin")
-    nanmax = _dtype_dispatch("nanmax")
-    nanmean = _dtype_dispatch("nanmean")
-    nanmedian = _dtype_dispatch("nanmedian")
-    nanstd = _dtype_dispatch("nanstd")
-    nanvar = _dtype_dispatch("nanvar")
+        def __call__(self, *args, **kwargs):
+            dt = args[0].dtype
+            if dt.kind == "f" and dt.itemsize == 8:
+                return bn_funcs[self.func_name](*args, **kwargs)
+            else:
+                return np_funcs[self.func_name](*args, **kwargs)
+
+    nansum = _DtypeDispatch("nansum")
+    nanmin = _DtypeDispatch("nanmin")
+    nanmax = _DtypeDispatch("nanmax")
+    nanmean = _DtypeDispatch("nanmean")
+    nanmedian = _DtypeDispatch("nanmedian")
+    nanstd = _DtypeDispatch("nanstd")
+    nanvar = _DtypeDispatch("nanvar")

 else:
     nansum = np.nansum


# BUG:  SigmaClip objects cannot be pickled when Bottleneck is installed

### Description

With `Bottleneck` installed:

```python
import pickle
from astropy.stats import SigmaClip

sigclip = SigmaClip(sigma=3.0, maxiters=5)
with open('sigclip.pkl', 'wb') as fh:
    pickle.dump(sigclip, fh)
```
error:
```
_pickle.PicklingError: Can't pickle local object <function _dtype_dispatch.<locals>.wrapped at 0x1096a43b0>
when serializing dict item '_cenfunc_parsed'
when serializing astropy.stats.sigma_clipping.SigmaClip state
when serializing astropy.stats.sigma_clipping.SigmaClip object
```

Git bisect points to this commit:  c06e4f7915bf889a64a84200e2d90cf23044bcae (https://github.com/astropy/astropy/pull/17204)

CC: @neutrinoceros

diff --git a/astropy/stats/tests/test_sigma_clipping.py b/astropy/stats/tests/test_sigma_clipping.py
index 9a5ee359fd01..c71d324c8f0d 100644
--- a/astropy/stats/tests/test_sigma_clipping.py
+++ b/astropy/stats/tests/test_sigma_clipping.py
@@ -1,5 +1,7 @@
 # Licensed under a 3-clause BSD style license - see LICENSE.rst

+import pickle
+
 import numpy as np
 import pytest
 from numpy.testing import assert_allclose, assert_equal
@@ -721,3 +723,18 @@ def test_propagation_of_mask():
     y = np.ma.masked_where(x > 1, x)

     assert_allclose(sigma_clipped_stats(y, grow=1), (1, 1, 0))
+
+
+def test_sigmaclip_pickle():
+    """Regression test for gh-19343: SigmaClip cannot be pickled
+    when Bottleneck is installed."""
+
+    sigclip = SigmaClip(sigma=3.0, maxiters=5)
+    restored = pickle.loads(pickle.dumps(sigclip))
+    assert restored.sigma == sigclip.sigma
+    assert restored.maxiters == sigclip.maxiters
+    # verify the restored object still works correctly
+    data = np.ma.array([1, 2, 3, 100, 4, 5])
+    result = restored(data)
+    expected = sigclip(data)
+    np.testing.assert_array_equal(result, expected)


# Fix CompImageHDU header when initializing from a PrimaryHDU header

### Description

This is a fix for https://github.com/astropy/astropy/issues/19361

@eigenbrot - would you be able to check if this does the trick for you?

- [x] By checking this box, the PR author has requested that maintainers do **NOT** use the "Squash and Merge" button. Maintainers should respect this when possible; however, the final decision is at the discretion of the maintainer that merges the PR.

diff --git a/astropy/io/fits/hdu/compressed/tests/test_compressed.py b/astropy/io/fits/hdu/compressed/tests/test_compressed.py
index d7d5ec6cb917..cb09477e3bea 100644
--- a/astropy/io/fits/hdu/compressed/tests/test_compressed.py
+++ b/astropy/io/fits/hdu/compressed/tests/test_compressed.py
@@ -1470,3 +1470,40 @@ def test_reserved_keywords_stripped(tmp_path):
         assert "ZBLANK" not in hdud[1].header
         assert "ZSCALE" not in hdud[1].header
         assert "ZZERO" not in hdud[1].header
+
+
+def test_compimghdu_with_primary_header_no_dual_keywords(tmp_path):
+    """
+    Regression test for https://github.com/astropy/astropy/issues/19361
+
+    When creating a CompImageHDU using a PrimaryHDU's header, the resulting
+    header should not contain both SIMPLE/ZSIMPLE and XTENSION/ZTENSION.
+    According to the FITS standard, a header must have either SIMPLE or
+    XTENSION, never both. Additionally, PCOUNT/GCOUNT are extension-only
+    keywords and should not appear in a primary-style header.
+    """
+    # Create a PrimaryHDU with some data
+    hdu = fits.PrimaryHDU(data=np.arange(100, dtype=np.int32).reshape(10, 10))
+
+    # Create a CompImageHDU using the PrimaryHDU's header
+    comp_hdu = fits.CompImageHDU(
+        data=np.arange(100, dtype=np.int32).reshape(10, 10),
+        header=hdu.header,
+    )
+
+    # The image header should have SIMPLE (not XTENSION) when created from PrimaryHDU
+    assert "SIMPLE" in comp_hdu.header
+    assert "XTENSION" not in comp_hdu.header
+    # PCOUNT/GCOUNT are extension-only keywords
+    assert "PCOUNT" not in comp_hdu.header
+    assert "GCOUNT" not in comp_hdu.header
+
+    # Write to file and check the raw bintable header
+    hdul = fits.HDUList([fits.PrimaryHDU(), comp_hdu])
+    hdul.writeto(tmp_path / "test.fits")
+
+    # Check the underlying bintable header has ZSIMPLE but not ZTENSION
+    with fits.open(tmp_path / "test.fits", disable_image_compression=True) as hdul:
+        bintable_header = hdul[1].header
+        assert "ZSIMPLE" in bintable_header
+        assert "ZTENSION" not in bintable_header

diff --git a/astropy/io/fits/hdu/compressed/compressed.py b/astropy/io/fits/hdu/compressed/compressed.py
index 8bac08aee0a1..52c21df0339a 100644
--- a/astropy/io/fits/hdu/compressed/compressed.py
+++ b/astropy/io/fits/hdu/compressed/compressed.py
@@ -335,7 +335,11 @@ def __init__(
             )

             if header is not None and "SIMPLE" in header:
-                self.header["SIMPLE"] = header["SIMPLE"]
+                # SIMPLE and XTENSION are mutually exclusive per FITS standard
+                del self.header["XTENSION"]
+                del self.header["PCOUNT"]
+                del self.header["GCOUNT"]
+                self.header.insert(0, ("SIMPLE", header["SIMPLE"]))

             self.compression_type = compression_type
             self.tile_shape = _validate_tile_shape(


# ECSV error reading if YAML's tag not used

Hi.

astropy.version.**version**: u'1.3.2'

I have a doubt whether the following behaviour is intentional.
In the following, *reading* means:
```
from astropy import table
t = table.Table.read('myfile.ecsv', format='ascii.ecsv')
```
Reading a ECSV file like the following crashes the process:
```
# %ECSV 0.9
# ---
# meta:
# - keyword:
#    this_is: a_test
# datatype:
# - name: fake
#   datatype: str
fake
0
```

And the problem is the missing argument `!!omap` after `meta`.
The followgin works fine:
```
# %ECSV 0.9
# ---
# meta: !!omap
# - keyword:
#    this_is: a_test
# datatype:
# - name: fake
#   datatype: str
fake
0
```

Is there a bug or the ordered dictionary (yaml's `omap`) is really mandatory for Astropy's table?
Thanks.

diff --git a/astropy/io/ascii/tests/test_ecsv.py b/astropy/io/ascii/tests/test_ecsv.py
index 8e7eaac8c203..adcf219e17ac 100644
--- a/astropy/io/ascii/tests/test_ecsv.py
+++ b/astropy/io/ascii/tests/test_ecsv.py
@@ -1305,3 +1305,77 @@ def test_compressed_files(tmp_path, format_engine, compressed_filename):
     # Open compressed file and compare to ensure it's read correctly
     t_comp = Table.read(compressed_filename, **format_engine)
     assert_array_equal(t, t_comp)
+
+
+def test_meta_not_omap_but_list(format_engine):
+    """The ecsv spec allows that header meta to be any valid safe YAML.
+
+    Though astropy's writer writes with an !!omap tag, and all examples
+    in the ecsv spec (https://github.com/astropy/astropy-APEs/blob/main/APE6.rst)
+    use that extra (non standard) tag, the spec specifically says
+    that the header meta
+    'an arbitrary data structure consisting purely of data types that can be encoded
+    and decoded with the YAML "safe" dumper and loader'.
+
+    See one example in https://github.com/astropy/astropy/issues/5990
+    """
+    txt = """
+# %ECSV 1.0
+# ---
+# meta:
+# - keywords:
+#   - {z_key1: val1}
+#   - {a_key2: val2}
+# - comments: [Comment 1, Comment 2, Comment 3]
+# datatype:
+# - name: fake
+#   datatype: string
+fake
+0"""
+    with pytest.warns(
+        UserWarning,
+        match="Found ECSV table meta of type list instead of",
+    ):
+        t = Table.read(txt, **format_engine)
+    assert len(t.meta) == 1
+    assert len(t.meta["meta"]) == 2
+    assert t.meta["meta"][0] == {"keywords": [{"z_key1": "val1"}, {"a_key2": "val2"}]}
+    assert t.meta["meta"][1] == {"comments": ["Comment 1", "Comment 2", "Comment 3"]}
+
+
+MAP_BUT_NOT_OMAP_LINES = [
+    "# %ECSV 1.0",
+    "# ---",
+    "# datatype:",
+    "# - {name: fake, datatype: string}",
+    "# meta: !!omap",
+    "# - {hr: 65}",
+    "# - {avg: 0.278}",
+    "# - {rbi: 147}",
+    "# schema: astropy-2.0",
+    "fake",
+    "0",
+]
+
+
+def test_meta_not_omap_but_map(format_engine):
+    """Like test_meta_not_omap_but_list but for a mapping"""
+    txt = """
+# %ECSV 1.0
+# ---
+# datatype:
+# - {name: fake, datatype: string}
+# meta:
+#   hr:  65    # Home runs
+#   avg: 0.278 # Batting average
+#   rbi: 147   # Runs Batted In
+# schema: astropy-2.0
+fake
+0"""
+    t = Table.read(txt, **format_engine)
+    assert t.meta["hr"] == 65
+    assert t.meta["avg"] == 0.278
+    assert t.meta["rbi"] == 147
+    out = StringIO()
+    t.write(out, format="ascii.ecsv")
+    assert out.getvalue().splitlines() == MAP_BUT_NOT_OMAP_LINES

diff --git a/astropy/io/ascii/ecsv.py b/astropy/io/ascii/ecsv.py
index a4992bc1eef9..0d088e289c18 100644
--- a/astropy/io/ascii/ecsv.py
+++ b/astropy/io/ascii/ecsv.py
@@ -13,6 +13,7 @@
 import numpy as np

 from astropy.io.ascii.core import convert_numpy
+from astropy.io.misc.ecsv import table_meta_as_dict
 from astropy.table import meta, serialize
 from astropy.utils.data_info import serialize_context_as
 from astropy.utils.exceptions import AstropyUserWarning
@@ -194,8 +195,7 @@ def get_cols(self, lines):
                 "unable to parse yaml in meta header"
             ) from e

-        if "meta" in header:
-            self.table_meta = header["meta"]
+        self.table_meta = table_meta_as_dict(header)

         if "delimiter" in header:
             delimiter = header["delimiter"]
@@ -515,7 +515,6 @@ class Ecsv(basic.Basic):
     ----- -----
       001     2
       004     3
-
     """

     _format_name = "ecsv"
diff --git a/astropy/io/misc/ecsv.py b/astropy/io/misc/ecsv.py
index efca71b7ab9b..6eb9fc7e00c0 100644
--- a/astropy/io/misc/ecsv.py
+++ b/astropy/io/misc/ecsv.py
@@ -489,6 +489,39 @@ def get_header_lines(
     return lines, idx, n_empty, n_comment

+def table_meta_as_dict(header) -> dict:
+    """Ensure meta information is some type of dictionary.
+
+    The ECSV format allows arbitrary data types for the `meta` keyword
+    but astropy requires a mapping (dict) for the meta information in a table.
+    For non-dict ECSV meta, we need to put the structure into a dict.
+    If that happens, warn the user of the conversion.
+
+    Parameters
+    ----------
+    header : dict
+        The header dictionary parsed from the ECSV file.
+        This contains the "meta" key if meta information is in the file.
+
+    Returns
+    -------
+    dict : dict
+        A dictionary containing the meta information. If the input `meta` is already
+        a dict, it is returned as is. Otherwise, it is wrapped in a dict under the key 'meta'.
+    """
+    if "meta" not in header:
+        return {}
+    meta = header["meta"]
+    if isinstance(meta, dict):
+        return meta
+
+    warnings.warn(
+        f"Found ECSV table meta of type {type(meta).__name__} instead of mapping (dict)."
+        f"This data structure is being copied to the 'meta' key of the output table meta attribute."
+    )
+    return {"meta": meta}
+
+
 def get_csv_np_type_dtype_shape(
     datatype: str, subtype: str | None, name: str
 ) -> DerivedColumnProperties:
@@ -654,7 +687,7 @@ def read_header(
     except meta.YamlParseError as e:
         raise InconsistentTableError("unable to parse yaml in meta header") from e

-    table_meta = header.get("meta", None)
+    table_meta = table_meta_as_dict(header)

     delimiter = header.get("delimiter", " ")
     if delimiter not in DELIMITERS:


# TNULL should be ignored for floating point columns in FITS tables

I tried downloading the FITS (ascii) table from here:

http://vizier.u-strasbg.fr/viz-bin/VizieR-3?-source=J/A%2bA/587/A153/yso&-out.max=50&-out.form=HTML%20Table&-out.add=_r&-out.add=_RAJ,_DEJ&-sort=_r&-oc.form=sexa

and if I read it in with Astropy I get:

```
In [8]: hdulist[1].columns
Out[8]:
ColDefs(
    name = '_RAJ2000'; format = 'F10.6'; unit = 'deg'; start = 2
    name = '_DEJ2000'; format = 'F10.6'; unit = 'deg'; start = 13
    name = 'VISION'; format = 'A16'; start = 24
    name = 'RAJ2000'; format = 'F10.6'; unit = 'deg'; start = 41
    name = 'DEJ2000'; format = 'F10.6'; unit = 'deg'; start = 52
    name = 'Jmag'; format = 'F6.3'; unit = 'mag'; null = 'NaN'; start = 63
    name = 'e_Jmag'; format = 'F6.3'; unit = 'mag'; null = 'NaN'; start = 70
    name = 'Hmag'; format = 'F6.3'; unit = 'mag'; null = 'NaN'; start = 77
    name = 'e_Hmag'; format = 'F6.3'; unit = 'mag'; null = 'NaN'; start = 84
    name = 'Ksmag'; format = 'F6.3'; unit = 'mag'; null = 'NaN'; start = 91
    name = 'e_Ksmag'; format = 'F6.3'; unit = 'mag'; null = 'NaN'; start = 98
    name = 'Japer'; format = 'F5.2'; unit = 'arcsec'; null = 'NaN'; start = 105
    name = 'Haper'; format = 'F5.2'; unit = 'arcsec'; null = 'NaN'; start = 111
    name = 'Ksaper'; format = 'F5.2'; unit = 'arcsec'; null = 'NaN'; start = 117
    name = 'Simbad'; format = 'A6'; start = 123
)
```

There are two issues here:

* The FITS standard doesn't allow TNULL for F columns:

https://fits.gsfc.nasa.gov/standard40/fits_standard40aa.pdf (page 23)

* Even if we decided to relax this, the null parameter should be forced to be a float or ignored (not set to a 'null' string above)

diff --git a/astropy/io/fits/tests/test_hdulist.py b/astropy/io/fits/tests/test_hdulist.py
index 760c29746a1a..3fa47dd11444 100644
--- a/astropy/io/fits/tests/test_hdulist.py
+++ b/astropy/io/fits/tests/test_hdulist.py
@@ -785,7 +785,7 @@ def test_fromstring(filename):
                         for n in hdul[idx].data.names:
                             c1 = hdul[idx].data[n]
                             c2 = hdul2[idx].data[n]
-                            assert (c1 == c2).all()
+                            np.testing.assert_array_equal(c1, c2)
                     elif any(dim == 0 for dim in hdul[idx].data.shape) or any(
                         dim == 0 for dim in hdul2[idx].data.shape
                     ):
diff --git a/astropy/io/fits/tests/test_table.py b/astropy/io/fits/tests/test_table.py
index 0acafd773626..ce1d99385e2a 100644
--- a/astropy/io/fits/tests/test_table.py
+++ b/astropy/io/fits/tests/test_table.py
@@ -1893,8 +1893,8 @@ def test_fits_rec_column_access(self):
         tbdata = fits.getdata(self.data("ascii.fits"))
         for col in ("a", "b"):
             data = getattr(tbdata, col)
-            assert (data == tbdata.field(col)).all()
-            assert (data == tbdata[col]).all()
+            np.testing.assert_array_equal(data, tbdata.field(col))
+            np.testing.assert_array_equal(data, tbdata[col])

         # with VLA column
         col1 = fits.Column(
@@ -2565,7 +2565,7 @@ def test_blank_field_zero(self):
             h.seek(0)
             h.write(nulled)

-        with fits.open(self.temp("ascii_null.fits"), memmap=True) as f:
+        with fits.open(self.temp("ascii_null.fits")) as f:
             assert f[1].data[2][0] == 0

         # Test a float column with a null value set and blank fields.
@@ -2586,10 +2586,23 @@ def test_blank_field_zero(self):
             h.seek(0)
             h.write(nulled)

-        with fits.open(self.temp("ascii_null2.fits"), memmap=True) as f:
-            # (Currently it should evaluate to 0.0, but if a TODO in fitsrec is
-            # completed, then it should evaluate to NaN.)
-            assert f[1].data[2][0] == 0.0 or np.isnan(f[1].data[2][0])
+        with fits.open(self.temp("ascii_null2.fits")) as f:
+            assert np.isnan(f[1].data[2][0])
+
+        # Test a float column with a NaN value
+        a = np.arange(10, dtype=float)
+        a[5] = np.nan
+        table = fits.TableHDU.from_columns(
+            [
+                fits.Column(name="a1", array=a, format="F"),
+                fits.Column(name="a2", array=a, format="F", null=np.nan),
+            ]
+        )
+        table.writeto(self.temp("ascii_null3.fits"))
+
+        with fits.open(self.temp("ascii_null3.fits")) as f:
+            assert np.isnan(f[1].data["a1"][5])
+            assert np.isnan(f[1].data["a2"][5])

     def test_column_array_type_mismatch(self):
         """Regression test for https://aeon.stsci.edu/ssb/trac/pyfits/ticket/218"""

diff --git a/astropy/io/fits/fitsrec.py b/astropy/io/fits/fitsrec.py
index 32b8ec436424..29cf77b88030 100644
--- a/astropy/io/fits/fitsrec.py
+++ b/astropy/io/fits/fitsrec.py
@@ -878,11 +878,14 @@ def _convert_ascii(self, column, field):
         # array buffer.
         dummy = np.char.ljust(field, format.width)
         dummy = np.char.replace(dummy, encode_ascii("D"), encode_ascii("E"))
-        null_fill = encode_ascii(str(ASCIITNULL).rjust(format.width))

-        # Convert all fields equal to the TNULL value (nullval) to empty fields.
-        # TODO: These fields really should be converted to NaN or something else undefined.
-        # Currently they are converted to empty fields, which are then set to zero.
+        # Convert all fields equal to the TNULL value (nullval) to either NaN
+        # for float columns or 0 for the other fields.
+        if format.format in "DEF":
+            null_fill = "nan"
+        else:
+            null_fill = str(ASCIITNULL)
+        null_fill = encode_ascii(null_fill.rjust(format.width))
         dummy = np.where(np.char.strip(dummy) == nullval, null_fill, dummy)

         # always replace empty fields, see https://github.com/astropy/astropy/pull/5394


# BUG: numpy.char.chararray is deprecated

xref: https://github.com/numpy/numpy/pull/30605

this causes about 800 tests to fail, most of which correspond to a single failure mode where we check an object type with `isinstance` and can easily be handled (I'll open a patch in a sec).
The rest of them are not necessarily difficult to handle, but patching may require breaking changes since we may need to change the return type of public functions (possibly indirectly) in cases where we deliberately instantiate this class.

Related issue:
- #3862

Linked PRs:
- #19217
- #19218
- #19233
- #19267

diff --git a/astropy/io/fits/tests/test_table.py b/astropy/io/fits/tests/test_table.py
index f3bae3f523a4..c91ad65eaa26 100644
--- a/astropy/io/fits/tests/test_table.py
+++ b/astropy/io/fits/tests/test_table.py
@@ -9,7 +9,6 @@

 import numpy as np
 import pytest
-from numpy import char as chararray

 try:
     import objgraph
@@ -24,6 +23,7 @@
 from astropy.io.fits.verify import VerifyError
 from astropy.table import Table
 from astropy.units import Unit, UnitsWarning, UnrecognizedUnit
+from astropy.utils.compat import get_chararray
 from astropy.utils.exceptions import AstropyUserWarning

 from .conftest import FitsTestCase
@@ -156,7 +156,7 @@ def test_open(self, home_is_data):
         fd = fits.open(self.data("test0.fits"))

         # create some local arrays
-        a1 = chararray.array(["abc", "def", "xx"])
+        a1 = get_chararray(["abc", "def", "xx"])
         r1 = np.array([11.0, 12.0, 13.0], dtype=np.float32)

         # create a table from scratch, using a mixture of columns from existing
@@ -319,7 +319,7 @@ def test_ascii_table(self):

         # Test Start Column

-        a1 = chararray.array(["abcd", "def"])
+        a1 = get_chararray(["abcd", "def"])
         r1 = np.array([11.0, 12.0])
         c1 = fits.Column(name="abc", format="A3", start=19, array=a1)
         c2 = fits.Column(name="def", format="E", start=3, array=r1)
@@ -1970,7 +1970,7 @@ def test_string_column_padding(self):
             "p\x00\x00\x00\x00\x00\x00\x00\x00\x00"
         )

-        acol = fits.Column(name="MEMNAME", format="A10", array=chararray.array(a))
+        acol = fits.Column(name="MEMNAME", format="A10", array=get_chararray(a))
         ahdu = fits.BinTableHDU.from_columns([acol])
         assert ahdu.data.tobytes().decode("raw-unicode-escape") == s
         ahdu.writeto(self.temp("newtable.fits"))

diff --git a/astropy/io/fits/column.py b/astropy/io/fits/column.py
index 804adb07c95f..ff513628367f 100644
--- a/astropy/io/fits/column.py
+++ b/astropy/io/fits/column.py
@@ -14,9 +14,9 @@
 from textwrap import indent

 import numpy as np
-from numpy import char as chararray

 from astropy.utils import lazyproperty
+from astropy.utils.compat import chararray, get_chararray
 from astropy.utils.exceptions import AstropyUserWarning

 from .card import CARD_LENGTH, Card
@@ -696,14 +696,14 @@ def __init__(
         # input arrays can be just list or tuple, not required to be ndarray
         # does not include Object array because there is no guarantee
         # the elements in the object array are consistent.
-        if not isinstance(array, (np.ndarray, chararray.chararray, Delayed)):
+        if not isinstance(array, (np.ndarray, chararray, Delayed)):
             try:  # try to convert to a ndarray first
                 if array is not None:
                     array = np.array(array)
             except Exception:
                 try:  # then try to convert it to a strings array
                     itemsize = int(recformat[1:])
-                    array = chararray.array(array, itemsize=itemsize)
+                    array = get_chararray(array, itemsize=itemsize)
                 except ValueError:
                     # then try variable length array
                     # Note: This includes _FormatQ by inheritance
@@ -1388,7 +1388,7 @@ def _convert_to_valid_data_type(self, array):
                         fsize = dims[-1]
                     else:
                         fsize = np.dtype(format.recformat).itemsize
-                    return chararray.array(array, itemsize=fsize, copy=False)
+                    return get_chararray(array, itemsize=fsize, copy=False)
                 else:
                     return _convert_array(array, np.dtype(format.recformat))
             elif "L" in format:
@@ -2082,7 +2082,7 @@ def __new__(cls, input, dtype="S"):
             try:
                 # this handles ['abc'] and [['a','b','c']]
                 # equally, beautiful!
-                input = [chararray.array(x, itemsize=1) for x in input]
+                input = [get_chararray(x, itemsize=1) for x in input]
             except Exception:
                 raise ValueError(f"Inconsistent input data array: {input}")

@@ -2105,10 +2105,10 @@ def __setitem__(self, key, value):
         """
         if isinstance(value, np.ndarray) and value.dtype == self.dtype:
             pass
-        elif isinstance(value, chararray.chararray) and value.itemsize == 1:
+        elif isinstance(value, chararray) and value.itemsize == 1:
             pass
         elif self.element_dtype == "S":
-            value = chararray.array(value, itemsize=1)
+            value = get_chararray(value, itemsize=1)
         else:
             value = np.array(value, dtype=self.element_dtype)
         np.ndarray.__setitem__(self, key, value)
@@ -2262,7 +2262,7 @@ def _makep(array, descr_output, format, nrows=None):
             else:
                 rowval = [0] * data_output.max
         if format.dtype == "S":
-            data_output[idx] = chararray.array(encode_ascii(rowval), itemsize=1)
+            data_output[idx] = get_chararray(encode_ascii(rowval), itemsize=1)
         else:
             data_output[idx] = np.array(rowval, dtype=format.dtype)

diff --git a/astropy/io/fits/fitsrec.py b/astropy/io/fits/fitsrec.py
index 9787b618e32c..1bdba8b3e5dc 100644
--- a/astropy/io/fits/fitsrec.py
+++ b/astropy/io/fits/fitsrec.py
@@ -8,9 +8,9 @@
 from functools import reduce

 import numpy as np
-from numpy import char as chararray

 from astropy.utils import lazyproperty
+from astropy.utils.compat import chararray, get_chararray

 from .column import (
     _VLF,
@@ -831,7 +831,7 @@ def _convert_p(self, column, field, recformat):
                 dt = np.dtype(recformat.dtype + str(1))
                 arr_len = count * dt.itemsize
                 da = raw_data[offset : offset + arr_len].view(dt)
-                da = np.char.array(da.view(dtype=dt), itemsize=count)
+                da = get_chararray(da.view(dtype=dt), itemsize=count)
                 dummy[idx] = decode_ascii(da)
             else:
                 dt = np.dtype(recformat.dtype)
@@ -1204,7 +1204,7 @@ def _scale_back(self, update_heap_pointers=True):
                 if isinstance(self._coldefs, _AsciiColDefs):
                     self._scale_back_ascii(index, dummy, raw_field)
                 # binary table string column
-                elif isinstance(raw_field, chararray.chararray):
+                elif isinstance(raw_field, chararray):
                     self._scale_back_strings(index, dummy, raw_field)
                 # all other binary table columns
                 else:
@@ -1361,8 +1361,8 @@ def _get_recarray_field(array, key):
     # This is currently needed for backwards-compatibility and for
     # automatic truncation of trailing whitespace
     field = np.recarray.field(array, key)
-    if field.dtype.char in ("S", "U") and not isinstance(field, chararray.chararray):
-        field = field.view(chararray.chararray)
+    if field.dtype.char in ("S", "U") and not isinstance(field, chararray):
+        field = field.view(chararray)
     return field

diff --git a/astropy/io/fits/hdu/table.py b/astropy/io/fits/hdu/table.py
index 3121c8d49077..e9d2f2f293fe 100644
--- a/astropy/io/fits/hdu/table.py
+++ b/astropy/io/fits/hdu/table.py
@@ -11,7 +11,6 @@
 from contextlib import suppress

 import numpy as np
-from numpy import char as chararray

 # This module may have many dependencies on astropy.io.fits.column, but
 # astropy.io.fits.column has fewer dependencies overall, so it's easier to
@@ -37,6 +36,7 @@
 from astropy.io.fits.header import Header, _pad_length
 from astropy.io.fits.util import _is_int, _str_to_num, path_like
 from astropy.utils import lazyproperty
+from astropy.utils.compat import chararray

 from .base import DELAYED, ExtensionHDU, _ValidHDU

@@ -1489,7 +1489,7 @@ def _binary_table_byte_swap(data):
         formats.append(field_dtype)
         offsets.append(field_offset)

-        if isinstance(field, chararray.chararray):
+        if isinstance(field, chararray):
             continue

         # only swap unswapped
@@ -1507,7 +1507,7 @@ def _binary_table_byte_swap(data):
             coldata = data.field(idx)
             for c in coldata:
                 if (
-                    not isinstance(c, chararray.chararray)
+                    not isinstance(c, chararray)
                     and c.itemsize > 1
                     and c.dtype.str[0] in swap_types
                 ):
diff --git a/astropy/utils/compat/numpycompat.py b/astropy/utils/compat/numpycompat.py
index 91cbc3805225..5fd0231a4fc2 100644
--- a/astropy/utils/compat/numpycompat.py
+++ b/astropy/utils/compat/numpycompat.py
@@ -4,6 +4,8 @@
 earlier versions of Numpy.
 """

+import warnings
+
 import numpy as np

 from astropy.utils import minversion
@@ -19,6 +21,8 @@
     "NUMPY_LT_2_4",
     "NUMPY_LT_2_4_1",
     "NUMPY_LT_2_5",
+    "chararray",
+    "get_chararray",
 ]

 # TODO: It might also be nice to have aliases to these named for specific
@@ -36,3 +40,9 @@

 COPY_IF_NEEDED = False if NUMPY_LT_2_0 else None
+
+with warnings.catch_warnings():
+    warnings.filterwarnings("ignore", category=DeprecationWarning)
+
+    get_chararray = np.char.array
+    chararray = np.char.chararray


# io.fits.getdata() lower and upper keywords broken

astropy 1.0.2 broke the functionality of the `lower` and `upper` arguments to astropy.io.fits.getdata when reading binary fits tables.  Prior to 1.0.2, the returned FITS_rec object properly handled case insensitive column names, while allowing the lower/upper keywords to specify what should be used if those objects were cast to normal numpy.array objects or written back out.  Starting with astropy 1.0.2, this broke; using the `lower` or `upper` arguments resulted in _neither_ case working.  The code below demonstrates the problem:

```
import astropy
print astropy.__version__

from astropy.table import Table
t = Table([np.zeros(5), np.ones(5), np.arange(5)], names=['a', 'b', 'c'])
t.write('lower.fits')
t = Table([np.zeros(5), np.ones(5), np.arange(5)], names=['A', 'B', 'C'])
t.write('upper.fits')

#- Without the lower/upper keywords, columns can be accessed with either
#- upper- or lower-case names; the .dtype.names reflects whatever was in
#- the source fits file
from astropy.io import fits
lower = fits.getdata('lower.fits', 1)
upper = fits.getdata('upper.fits', 1)

print lower.dtype.names
print upper.dtype.names

np.all(lower['a'] == lower['A'])    #- Works
np.all(upper['a'] == upper['A'])    #- Works

#- Prior to astropy 1.0.2, the upper/lower keywords affected the case of
#- .dtype.names and how they are cast to normal numpy.array objects, while
#- still supporting case-insensitive column names.  This broke with
#- astropy 1.0.2, where neither upper- nor lower-case works after using
#- the upper/lower keywords.
lowup = fits.getdata('lower.fits', 1, upper=True)
lowup.dtype.names
lowup['A']              #- KeyError
lowup['a']              #- KeyError
lowup.A                 #- Works
np.array(lowup)['A']    #- Works

uplow = fits.getdata('upper.fits', 1, lower=True)
uplow.dtype.names
uplow['A']              #- KeyError
uplow['a']              #- KeyError
uplow.a                 #- Works
np.array(uplow)['a']    #- Works
```

diff --git a/astropy/io/fits/tests/test_convenience.py b/astropy/io/fits/tests/test_convenience.py
index 14c021dadea2..a699f0b30534 100644
--- a/astropy/io/fits/tests/test_convenience.py
+++ b/astropy/io/fits/tests/test_convenience.py
@@ -513,3 +513,22 @@ def test_getdata_ext_not_given_nodata_noext(self):
             IndexError, match="No data in Primary HDU and no extension HDU found."
         ):
             fits.getdata(buf)
+
+    def test_getdata_lower_upper(self, tmp_path):
+        t = Table([np.zeros(5), np.ones(5), np.arange(5)], names=["a", "b", "c"])
+        t.write(tmp_path / "lower.fits")
+        t = Table([np.zeros(5), np.ones(5), np.arange(5)], names=["A", "B", "C"])
+        t.write(tmp_path / "upper.fits")
+
+        ref = np.zeros(5)
+        lowup = fits.getdata(tmp_path / "lower.fits", 1, upper=True)
+        assert lowup.dtype.names == ("A", "B", "C")
+        assert_array_equal(lowup["A"], ref)
+        assert_array_equal(lowup["a"], ref)
+        assert_array_equal(lowup.A, ref)
+
+        uplow = fits.getdata(tmp_path / "upper.fits", 1, lower=True)
+        assert uplow.dtype.names == ("a", "b", "c")
+        assert_array_equal(uplow["A"], ref)
+        assert_array_equal(uplow["a"], ref)
+        assert_array_equal(uplow.a, ref)

diff --git a/astropy/io/fits/convenience.py b/astropy/io/fits/convenience.py
index 3253ba9caa2f..83b64816071d 100644
--- a/astropy/io/fits/convenience.py
+++ b/astropy/io/fits/convenience.py
@@ -251,7 +251,9 @@ def getdata(filename, *args, header=None, lower=None, upper=None, view=None, **k
         if data.dtype.descr[0][0] == "":
             # this data does not have fields
             return
-        data.dtype.names = [trans(n) for n in data.dtype.names]
+
+        for n in data.dtype.names:
+            data.columns.change_name(n, trans(n))

     # allow different views into the underlying ndarray.  Keep the original
     # view just in case there is a problem


# Fix getdata()'s lower and upper keywords.

Fix #4105

<!-- These comments are hidden when you submit the pull request,
so you do not need to remove them! -->

<!-- Please be sure to check out our contributing guidelines,
https://github.com/astropy/astropy/blob/main/CONTRIBUTING.md .
Please be sure to check out our code of conduct,
https://github.com/astropy/astropy/blob/main/CODE_OF_CONDUCT.md . -->

<!-- If you are new or need to be re-acquainted with Astropy
contributing workflow, please see
https://docs.astropy.org/en/latest/development/quickstart.html .
There is even a practical example at
https://docs.astropy.org/en/latest/development/git_edit_workflow_examples.html . -->

<!-- Please just have a quick search on GitHub to see if a similar
pull request has already been posted.
We have old closed pull requests that might provide useful code or ideas
that directly tie in with your pull request. -->

<!-- We have several automatic features that run when a pull request is open.
They can appear daunting but do not worry because maintainers will help
you navigate them, if necessary. -->

### Description
<!-- Provide a general description of what your pull request does.
Complete the following sentence and add relevant details as you see fit. -->

<!-- In addition please ensure that the pull request title is descriptive
and allows maintainers to infer the applicable subpackage(s). -->

<!-- READ THIS FOR MANUAL BACKPORT FROM A MAINTAINER:
Apply "skip-basebranch-check" label **before** you open the PR! -->

This pull request is to address ...

<!-- If the pull request closes any open issues you can add this.
If you replace <Issue Number> with a number, GitHub will automatically link it.
If this pull request is unrelated to any issues, please remove
the following line. -->

Fixes #<Issue Number>

<!-- Optional opt-out -->

- [ ] By checking this box, the PR author has requested that maintainers do **NOT** use the "Squash and Merge" button. Maintainers should respect this when possible; however, the final decision is at the discretion of the maintainer that merges the PR.


# Deprecate use of chararray in io.fits

As raised in #3854 and earlier issues, Numpy is gradually trying to deprecate and eventually kill off the old Numarray `chararray` class.  This is fine, but `io.fits` still uses it for FITS tables.  This is especially important in supporting FITS ASCII tables, for its ability to automatically `rstrip()` individual strings returned from the array.

However, for FITS data we can just use our own class that has this capability (we don't need any of the other `chararray` methods).  It might be nice to go ahead and use some version of my [`encoded_text_array`](https://github.com/embray/PyFITS/blob/c11abd331cff7e177f0fc551188347e529cde92d/lib/pyfits/fitsrec.py#L999) class for FITS string columns (and possible string columns in other Astropy tables), and to add _optional_ rstrip support to that class (it already has automatic rstripping, but that should probably be optional).

I think for backwards compat the new class _should_ inherit `chararray` methods, but they would all raise deprecation warnings.  Similarly `isinstance(..., chararray)` checks should raise a deprecation warning (that the class will no longer be a chararray subclass after long).

diff --git a/astropy/io/fits/tests/test_table.py b/astropy/io/fits/tests/test_table.py
index 3196fe9ab294..61acb9812d97 100644
--- a/astropy/io/fits/tests/test_table.py
+++ b/astropy/io/fits/tests/test_table.py
@@ -9,7 +9,6 @@

 import numpy as np
 import pytest
-from numpy import char as chararray

 try:
     import objgraph
@@ -24,7 +23,8 @@
 from astropy.io.fits.verify import VerifyError
 from astropy.table import Table
 from astropy.units import Unit, UnitsWarning, UnrecognizedUnit
-from astropy.utils.exceptions import AstropyUserWarning
+from astropy.utils.compat import get_chararray
+from astropy.utils.exceptions import AstropyDeprecationWarning, AstropyUserWarning

 from .conftest import FitsTestCase
 from .test_connect import TestMultipleHDU
@@ -156,7 +156,7 @@ def test_open(self, home_is_data):
         fd = fits.open(self.data("test0.fits"))

         # create some local arrays
-        a1 = chararray.array(["abc", "def", "xx"])
+        a1 = get_chararray(["abc", "def", "xx"])
         r1 = np.array([11.0, 12.0, 13.0], dtype=np.float32)

         # create a table from scratch, using a mixture of columns from existing
@@ -319,7 +319,7 @@ def test_ascii_table(self):

         # Test Start Column

-        a1 = chararray.array(["abcd", "def"])
+        a1 = get_chararray(["abcd", "def"])
         r1 = np.array([11.0, 12.0])
         c1 = fits.Column(name="abc", format="A3", start=19, array=a1)
         c2 = fits.Column(name="def", format="E", start=3, array=r1)
@@ -1970,7 +1970,7 @@ def test_string_column_padding(self):
             "p\x00\x00\x00\x00\x00\x00\x00\x00\x00"
         )

-        acol = fits.Column(name="MEMNAME", format="A10", array=chararray.array(a))
+        acol = fits.Column(name="MEMNAME", format="A10", array=get_chararray(a))
         ahdu = fits.BinTableHDU.from_columns([acol])
         assert ahdu.data.tobytes().decode("raw-unicode-escape") == s
         ahdu.writeto(self.temp("newtable.fits"))
@@ -2228,8 +2228,14 @@ def test_new_table_with_nd_column(self):
         with fits.open(self.temp("test.fits")) as h:
             # Need to force string arrays to byte arrays in order to compare
             # correctly on Python 3
-            assert (h[1].data["str"].encode("ascii") == arra).all()
-            assert (h[1].data["strarray"].encode("ascii") == arrb).all()
+            with pytest.warns(
+                AstropyDeprecationWarning, match="chararray is deprecated.*"
+            ):
+                assert (h[1].data["str"].encode("ascii") == arra).all()
+            with pytest.warns(
+                AstropyDeprecationWarning, match="chararray is deprecated.*"
+            ):
+                assert (h[1].data["strarray"].encode("ascii") == arrb).all()
             assert (h[1].data["intarray"] == arrc).all()

     def test_mismatched_tform_and_tdim(self):

diff --git a/astropy/io/fits/column.py b/astropy/io/fits/column.py
index 804adb07c95f..ff513628367f 100644
--- a/astropy/io/fits/column.py
+++ b/astropy/io/fits/column.py
@@ -14,9 +14,9 @@
 from textwrap import indent

 import numpy as np
-from numpy import char as chararray

 from astropy.utils import lazyproperty
+from astropy.utils.compat import chararray, get_chararray
 from astropy.utils.exceptions import AstropyUserWarning

 from .card import CARD_LENGTH, Card
@@ -696,14 +696,14 @@ def __init__(
         # input arrays can be just list or tuple, not required to be ndarray
         # does not include Object array because there is no guarantee
         # the elements in the object array are consistent.
-        if not isinstance(array, (np.ndarray, chararray.chararray, Delayed)):
+        if not isinstance(array, (np.ndarray, chararray, Delayed)):
             try:  # try to convert to a ndarray first
                 if array is not None:
                     array = np.array(array)
             except Exception:
                 try:  # then try to convert it to a strings array
                     itemsize = int(recformat[1:])
-                    array = chararray.array(array, itemsize=itemsize)
+                    array = get_chararray(array, itemsize=itemsize)
                 except ValueError:
                     # then try variable length array
                     # Note: This includes _FormatQ by inheritance
@@ -1388,7 +1388,7 @@ def _convert_to_valid_data_type(self, array):
                         fsize = dims[-1]
                     else:
                         fsize = np.dtype(format.recformat).itemsize
-                    return chararray.array(array, itemsize=fsize, copy=False)
+                    return get_chararray(array, itemsize=fsize, copy=False)
                 else:
                     return _convert_array(array, np.dtype(format.recformat))
             elif "L" in format:
@@ -2082,7 +2082,7 @@ def __new__(cls, input, dtype="S"):
             try:
                 # this handles ['abc'] and [['a','b','c']]
                 # equally, beautiful!
-                input = [chararray.array(x, itemsize=1) for x in input]
+                input = [get_chararray(x, itemsize=1) for x in input]
             except Exception:
                 raise ValueError(f"Inconsistent input data array: {input}")

@@ -2105,10 +2105,10 @@ def __setitem__(self, key, value):
         """
         if isinstance(value, np.ndarray) and value.dtype == self.dtype:
             pass
-        elif isinstance(value, chararray.chararray) and value.itemsize == 1:
+        elif isinstance(value, chararray) and value.itemsize == 1:
             pass
         elif self.element_dtype == "S":
-            value = chararray.array(value, itemsize=1)
+            value = get_chararray(value, itemsize=1)
         else:
             value = np.array(value, dtype=self.element_dtype)
         np.ndarray.__setitem__(self, key, value)
@@ -2262,7 +2262,7 @@ def _makep(array, descr_output, format, nrows=None):
             else:
                 rowval = [0] * data_output.max
         if format.dtype == "S":
-            data_output[idx] = chararray.array(encode_ascii(rowval), itemsize=1)
+            data_output[idx] = get_chararray(encode_ascii(rowval), itemsize=1)
         else:
             data_output[idx] = np.array(rowval, dtype=format.dtype)

diff --git a/astropy/io/fits/connect.py b/astropy/io/fits/connect.py
index 1439c62b6caf..6643ab1ee54a 100644
--- a/astropy/io/fits/connect.py
+++ b/astropy/io/fits/connect.py
@@ -295,7 +295,7 @@ def read_table_fits(
         coltype = col.dtype.subdtype[0].type if col.dtype.subdtype else col.dtype.type

         if strip_spaces and coltype is np.bytes_:
-            arr = arr.rstrip()
+            arr = np.strings.rstrip(arr)

         # Check if column is masked. Here, we make a guess based on the
         # presence of FITS mask values. For integer columns, this is simply
diff --git a/astropy/io/fits/fitsrec.py b/astropy/io/fits/fitsrec.py
index 6f5e0f511cc1..32b8ec436424 100644
--- a/astropy/io/fits/fitsrec.py
+++ b/astropy/io/fits/fitsrec.py
@@ -8,9 +8,9 @@
 from functools import reduce

 import numpy as np
-from numpy import char as chararray

 from astropy.utils import lazyproperty
+from astropy.utils.compat import chararray, get_chararray

 from .column import (
     _VLF,
@@ -831,7 +831,7 @@ def _convert_p(self, column, field, recformat):
                 dt = np.dtype(recformat.dtype + str(1))
                 arr_len = count * dt.itemsize
                 da = raw_data[offset : offset + arr_len].view(dt)
-                da = np.char.array(da.view(dtype=dt), itemsize=count)
+                da = get_chararray(da.view(dtype=dt), itemsize=count)
                 dummy[idx] = decode_ascii(da)
             else:
                 dt = np.dtype(recformat.dtype)
@@ -1204,7 +1204,7 @@ def _scale_back(self, update_heap_pointers=True):
                 if isinstance(self._coldefs, _AsciiColDefs):
                     self._scale_back_ascii(index, dummy, raw_field)
                 # binary table string column
-                elif isinstance(raw_field, chararray.chararray):
+                elif isinstance(raw_field, chararray):
                     self._scale_back_strings(index, dummy, raw_field)
                 # all other binary table columns
                 else:
@@ -1341,7 +1341,7 @@ def _scale_back_ascii(self, col_idx, input_field, output_field):

         # Replace exponent separator in floating point numbers
         if "D" in format:
-            output_field[:] = output_field.replace(b"E", b"D")
+            output_field[:] = np.strings.replace(output_field, b"E", b"D")

     def tolist(self):
         # Override .tolist to take care of special case of VLF
@@ -1361,8 +1361,8 @@ def _get_recarray_field(array, key):
     # This is currently needed for backwards-compatibility and for
     # automatic truncation of trailing whitespace
     field = np.recarray.field(array, key)
-    if field.dtype.char in ("S", "U") and not isinstance(field, chararray.chararray):
-        field = field.view(chararray.chararray)
+    if field.dtype.char in ("S", "U") and not isinstance(field, chararray):
+        field = field.view(chararray)
     return field

diff --git a/astropy/io/fits/hdu/table.py b/astropy/io/fits/hdu/table.py
index 16528f813fb7..362dc2cd8ee9 100644
--- a/astropy/io/fits/hdu/table.py
+++ b/astropy/io/fits/hdu/table.py
@@ -11,7 +11,6 @@
 from contextlib import suppress

 import numpy as np
-from numpy import char as chararray

 # This module may have many dependencies on astropy.io.fits.column, but
 # astropy.io.fits.column has fewer dependencies overall, so it's easier to
@@ -37,6 +36,7 @@
 from astropy.io.fits.header import Header, _pad_length
 from astropy.io.fits.util import _is_int, _str_to_num, path_like
 from astropy.utils import lazyproperty
+from astropy.utils.compat import chararray

 from .base import DELAYED, ExtensionHDU, _ValidHDU

@@ -1489,7 +1489,7 @@ def _binary_table_byte_swap(data):
         formats.append(field_dtype)
         offsets.append(field_offset)

-        if isinstance(field, chararray.chararray):
+        if isinstance(field, chararray):
             continue

         # only swap unswapped
@@ -1507,7 +1507,7 @@ def _binary_table_byte_swap(data):
             coldata = data.field(idx)
             for c in coldata:
                 if (
-                    not isinstance(c, chararray.chararray)
+                    not isinstance(c, chararray)
                     and c.itemsize > 1
                     and c.dtype.str[0] in swap_types
                 ):
diff --git a/astropy/utils/compat/numpycompat.py b/astropy/utils/compat/numpycompat.py
index dde37ec3701d..b4f40fb1caa9 100644
--- a/astropy/utils/compat/numpycompat.py
+++ b/astropy/utils/compat/numpycompat.py
@@ -9,7 +9,10 @@
 import numpy as np

 from astropy.utils import minversion
-from astropy.utils.exceptions import AstropyPendingDeprecationWarning
+from astropy.utils.exceptions import (
+    AstropyDeprecationWarning,
+    AstropyPendingDeprecationWarning,
+)

 __all__ = [
     "NUMPY_LT_2_1",
@@ -18,6 +21,8 @@
     "NUMPY_LT_2_4",
     "NUMPY_LT_2_4_1",
     "NUMPY_LT_2_5",
+    "chararray",
+    "get_chararray",
 ]

 # TODO: It might also be nice to have aliases to these named for specific
@@ -42,3 +47,43 @@ def __getattr__(attr):
         return None

     raise AttributeError(f"module {__name__!r} has no attribute {attr!r}.")
+
+
+with warnings.catch_warnings():
+    warnings.filterwarnings("ignore", category=DeprecationWarning)
+
+    from numpy.char import array as np_char_array
+    from numpy.char import chararray as np_chararray
+
+
+_deprecated_chararray_attributes = {
+    "capitalize", "center", "count", "decode", "encode", "endswith",
+    "expandtabs", "find", "index", "isalnum", "isalpha", "isdecimal",
+    "isdigit", "islower", "isnumeric", "isspace", "istitle", "isupper",
+    "join", "ljust", "lower", "lstrip", "replace", "rfind", "rindex",
+    "rjust", "rpartition", "rsplit", "rstrip", "split", "splitlines",
+    "startswith", "strip", "swapcase", "title", "translate", "upper",
+    "zfill"
+}  # fmt: skip
+
+
+class chararray(np_chararray):
+    """Version of np.char.chararray with deprecation warnings on special methods."""
+
+    def __getattribute__(self, name):
+        if name in _deprecated_chararray_attributes:
+            warnings.warn(
+                "chararray is deprecated, in future versions astropy will "
+                "return a normal array so the special chararray methods "
+                "(e.g., .rstrip()) will not be available. Use np.strings "
+                "functions instead.",
+                AstropyDeprecationWarning,
+            )
+        return super().__getattribute__(name)
+
+
+def get_chararray(obj, itemsize=None, copy=True, unicode=None, order=None):
+    """Get version of np.char.chararray that gives deprecation warnings on special methods."""
+    return np_char_array(
+        obj, itemsize=itemsize, copy=copy, unicode=unicode, order=order
+    ).view(chararray)