Fixed #36644 -- Enabled empty order_by() to avoid pk ordering by first()/last().

#### Trac ticket number
<!-- Replace XXXXX with the corresponding Trac ticket number, or delete the line and write "N/A" if this is a trivial PR. -->

ticket-36644

#### Branch description
Enabled calling `order_by()` without arguments to disable implicit primary key ordering in `first()`/`last()`. When a queryset is explicitly marked as unordered via `order_by()`, `first()`/`last()` now respect this and avoid adding PK ordering. Also reset default_ordering in `_combinator_query()` so that `union().first()` can still add pk ordering.

Discussion: https://github.com/[USER_REDACTED]ngo/new-features/issues/71,

#### Checklist
- [x] This PR targets the `main` branch. <!-- Backports will be evaluated and done by mergers, when necessary. -->
- [x] The commit message is written in past tense, mentions the ticket number, and ends with a period.
- [x] I have checked the "Has patch" ticket flag in the Trac system.
- [x] I have added or updated relevant tests.
- [ ] I have added or updated relevant docs, including release notes if applicable.
- [ ] I have attached screenshots in both light and dark modes for any UI changes.

@@ -1158,7 +1158,7 @@ async def alatest(self, *fields):

     def first(self):
         """Return the first object of a query or None if no match is found."""
-        if self.ordered:
+        if self.ordered or not self.query.default_ordering:
             queryset = self
         else:
             self._check_ordering_first_last_queryset_aggregation(method="first")
@@ -1171,7 +1171,7 @@ async def afirst(self):

     def last(self):
         """Return the last object of a query or None if no match is found."""
-        if self.ordered:
+        if self.ordered or not self.query.default_ordering:
             queryset = self.reverse()
         else:
             self._check_ordering_first_last_queryset_aggregation(method="last")
@@ -1679,6 +1679,7 @@ def _combinator_query(self, combinator, *other_qs, all=False):
         clone = self._chain()
         # Clear limits and ordering so they can be reapplied
         clone.query.clear_ordering(force=True)
+        clone.query.default_ordering = True
         clone.query.clear_limits()
         clone.query.combined_queries = (self.query, *(qs.query for qs in other_qs))
         clone.query.combinator = combinator

@@ -21,6 +21,14 @@ class Comment(models.Model):
     likes_count = models.PositiveIntegerField()

+class OrderedArticle(models.Model):
+    headline = models.CharField(max_length=100)
+    pub_date = models.DateField()
+
+    class Meta:
+        ordering = ["headline"]
+
+
 # Ticket #23555 - model with an intentionally broken QuerySet.__iter__ method.

@@ -1,9 +1,10 @@
 from datetime import datetime
+from unittest.mock import patch

 from django.db.models import Avg
 from django.test import TestCase

-from .models import Article, Comment, IndexErrorArticle, Person
+from .models import Article, Comment, IndexErrorArticle, OrderedArticle, Person

 class EarliestOrLatestTests(TestCase):
@@ -265,3 +266,30 @@ def test_first_last_unordered_qs_aggregation_error(self):
             qs.first()
         with self.assertRaisesMessage(TypeError, msg % "last"):
             qs.last()
+
+    def test_first_last_empty_order_by_has_no_pk_ordering(self):
+        Article.objects.create(
+            headline="Article 1",
+            pub_date=datetime(2006, 9, 10),
+            expire_date=datetime(2056, 9, 11),
+        )
+
+        qs = Article.objects.order_by()
+        with patch.object(type(qs), "order_by") as mock_order_by:
+            qs.first()
+            mock_order_by.assert_not_called()
+            qs.last()
+            mock_order_by.assert_not_called()
+
+    def test_first_last_empty_order_by_clears_default_ordering(self):
+        OrderedArticle.objects.create(
+            headline="Article 1",
+            pub_date=datetime(2006, 9, 10),
+        )
+
+        qs = OrderedArticle.objects.order_by()
+        with patch.object(type(qs), "order_by") as mock_order_by:
+            qs.first()
+            mock_order_by.assert_not_called()
+            qs.last()
+            mock_order_by.assert_not_called()

@@ -418,6 +418,15 @@ def test_union_with_first(self):
         qs2 = base_qs.filter(name="a2")
         self.assertEqual(qs1.union(qs2).first(), a1)

+    [USER_REDACTED]("supports_slicing_ordering_in_compound")
+    def test_union_applies_default_ordering_afterward(self):
+        c = Tag.objects.create(name="C")
+        Tag.objects.create(name="B")
+        a = Tag.objects.create(name="A")
+        qs1 = Tag.objects.filter(name__in=["A", "B"])[:1]
+        qs2 = Tag.objects.filter(name__in=["C"])[:1]
+        self.assertSequenceEqual(qs1.union(qs2), [a, c])
+
     def test_union_multiple_models_with_values_list_and_order(self):
         reserved_name = ReservedName.objects.create(name="rn1", order=0)
         qs1 = Celebrity.objects.all()

Thanks [USER_REDACTED] 👍

I have initial comments for you.

Thanks [USER_REDACTED], this looks great ✅ .

I segregated out the `union()` stuff since that was a separate bug we had to solve first. I also rewrote your test to actually perform queries and more clearly demonstrate that default orders were not applied after `union()`.

Need release notes and deprecation entries, see https://docs.djangoproject.com/en/6.0/internals/contributing/writing-code/submitting-patches/#deprecating-a-feature.

first() and last() returning the same object because only one object exists is a little non-intuitive for me when reading this test.

Can you add another test method showing what happens when a model has a default ordering as mentioned in the ticket?

I can deindent this without any failures.

[USER_REDACTED] the main reason I suggested the deprecation path was because of this test case.

Looking again, perhaps we can look through the other end of the telescope here and make `first()`/`last()` have no change for union()/combinators instead.

The idea would be:
```py
.order_by().first()  # no pk ordering, forced order clear
# teach .union() to forget about whatever forced order clearing came before
qs2.order_by().union(qs1).first()  # still has a pk ordering, since we forgot about the forced order clear before union()
qs2.order_by().union(qs1).order_by().first()  # no pk ordering, ordering was force-cleared after union()
```
WDYT?

[USER_REDACTED] you might want to pause on polishing the deprecation path to allow a little time for this discussion :-)

Hmm, interesting.

From an SQL point of view it looks like there's only ever an ordering applied to the whole union not to sub-parts (unless I'm missing something). In that case, I'm happy with your proposed behaviour.

It might also be worthwhile to warn about explicit `order_by` calls on either of the `union` querysets (i.e. `qs1` and `qs2` in your example), since these will always be ignored.

On the point of deprecation, I'm happy for the default behaviour to be a deprecation warning, but I'd like to also have a way to opt-in to the future behaviour (via a setting or maybe an attribute on the queryset or something).

Cool. So in that case, we don't need deprecation warnings relating to the order_by() <--> first() interaction, because we found a way to solve the ticket without mucking with this union() test case.

A tangential finding here about the order_by() <--> union() interaction might be worth a deprecation path to eventually lead to raising an error, but we don't need to tackle that here.

Does that make sense [USER_REDACTED]?

> A tangential finding here about the order_by() <--> union() interaction might be worth a deprecation path to eventually lead to raising an error, but we don't need to tackle that here.

Actually, we don't even need that, the reason these examples have `order_by()` before `.union()` is to clear a default ordering from the queryset, which is good, because otherwise `.union()` throws. So nothing to deprecate and nothing to raise an error for.

The question here is just how to disambiguate that from "please don't order my final result after the union".

I think `qs2.union(qs1).first()` should be "apply pk ordering" and `qs2.union(qs1).order_by().first()` should be "apply no ordering". (With extra `.order_by()` on `qs1` or `qs2` as necessary.)

Wonderful, I agree. So [USER_REDACTED], we should leave the original test case as is to capture Lily's first case, and then add an assertion to capture Lily's second case.

This won't fail if the code change is undone; we'll need something else, e.g. mocking `order_by()` and calling `assert_not_called`.

Thanks for the review! I’m aware of this and already working on the suggested changes. I’ll request another review once everything is ready.

No problem, sorry for jumping the gun 👍


Added DatabaseFeatures.supports_inspectdb.

Needed by MongoDB.

@@ -44,10 +44,10 @@ def add_arguments(self, parser):
         )

     def handle(self, **options):
-        try:
+        if connections[options["database"]].features.supports_inspectdb:
             for line in self.handle_inspection(options):
                 self.stdout.write(line)
-        except NotImplementedError:
+        else:
             raise CommandError(
                 "Database inspection isn't supported for the currently selected "
                 "database backend."

@@ -400,6 +400,9 @@ class BaseDatabaseFeatures:
     supports_on_delete_db_default = True
     supports_on_delete_db_null = True

+    # Does the backend support the inspectdb management command?
+    supports_inspectdb = True
+
     # Collation names for use by [NAME_REDACTED]
     test_collations = {
         "ci": None,  # Case-insensitive.

@@ -367,6 +367,9 @@ backends.
   ``db_on_delete``) as values. ``db_on_delete`` is one of the database-level
   delete options e.g. :attr:`~django.db.models.DB_CASCADE`.

+* Set the new ``DatabaseFeatures.supports_inspectdb`` attribute to ``False``
+  if the management command isn't supported.
+
 :mod:`django.contrib.gis`
 -------------------------

@@ -14,6 +14,7 @@
 from .models import AllOGRFields

+[USER_REDACTED]("supports_inspectdb")
 class InspectDbTests(TestCase):
     def test_geom_columns(self):
         """

@@ -2,11 +2,16 @@
 from io import StringIO
 from unittest import mock, skipUnless

-from django.core.management import call_command
+from django.core.management import CommandError, call_command
 from django.core.management.commands import inspectdb
 from django.db import connection
 from django.db.backends.base.introspection import TableInfo
-from django.test import TestCase, TransactionTestCase, skipUnlessDBFeature
+from django.test import (
+    TestCase,
+    TransactionTestCase,
+    skipIfDBFeature,
+    skipUnlessDBFeature,
+)

 from .models import PeopleMoreData, test_collation

@@ -40,6 +45,7 @@ def cursor_execute(*queries):
     return results

+[USER_REDACTED]("supports_inspectdb")
 class InspectDBTestCase(TestCase):
     unique_re = re.compile(r".*unique_together = \((.+),\).*")

@@ -519,6 +525,7 @@ def test_same_relations(self):
         )

+[USER_REDACTED]("supports_inspectdb")
 class InspectDBTransactionalTests(TransactionTestCase):
     available_apps = ["inspectdb"]

@@ -671,3 +678,14 @@ def test_composite_primary_key_not_unique_together(self):
         out = StringIO()
         call_command("inspectdb", "inspectdb_compositepkmodel", stdout=out)
         self.assertNotIn("unique_together", out.getvalue())
+
+
+[USER_REDACTED]("supports_inspectdb")
+class InspectDBNotSupportedTests(TestCase):
+    def test_not_supported(self):
+        msg = (
+            "Database inspection isn't supported for the currently selected "
+            "database backend."
+        )
+        with self.assertRaisesMessage(CommandError, msg):
+            call_command("inspectdb")

This matches what `inspectdb` does:
https://github.com/[USER_REDACTED]ngo/django/blob/68859a9b45f0b9d472c9277dc61e54afacc0da38/django/core/management/commands/inspectdb.py#L47-L54
I debated whether to use this implementation or to make it a boolean that the developer has to set to opt out. I could be convinced either way.

Interesting. I checked the blame, and at the time this was added the only backend that raised `NotImplementedError` from `get_table_names()` was the since-removed backend for ADO MSSQL:

https://github.com/[USER_REDACTED]ngo/django/blob/6d210ef40341b26aa2881fbb82ef7a7cdd54a3df/django/core/db/backends/ado_mssql.py#L112-L113

We could probably make the flag a boolean and then do a cleanup in `inspectdb` to replace this except with a check on that flag. wdyt?


Bumped linter dependencies

#### Trac ticket number

None. I can create one if we think it's necessary.

#### Branch description
Updates the linters.

* psf/black-pre-commit-mirror: 25.9.0 -> 25.12.0
* pre-commit/mirrors-eslint: v9.36.0 -> v9.39.1
* zizmorcore/zizmor-pre-commit: v1.16.3 -> v1.19.0
* zizmorcore/zizmor-action: 0.2.0 -> 0.3.0

#### Checklist
- [x] This PR targets the `main` branch. <!-- Backports will be evaluated and done by mergers, when necessary. -->
- [x] The commit message is written in past tense, mentions the ticket number, and ends with a period.
- [ ] I have checked the "Has patch" ticket flag in the Trac system.
- [ ] I have added or updated relevant tests.
- [ ] I have added or updated relevant docs, including release notes if applicable.
- [ ] I have attached screenshots in both light and dark modes for any UI changes.

@@ -79,7 +79,7 @@ jobs:
         with:
           persist-credentials: false
       - name: Run zizmor
-        uses: zizmorcore/zizmor-action@e673c3917a1aef3c65c972347ed84ccd013ecda4 # v0.2.0
+        uses: zizmorcore/zizmor-action@e639db99335bc9038abc0e066dfcd72e23d26fb4 # v0.3.0
         with:
           advanced-security: false
           annotations: true

@@ -1,6 +1,6 @@
 repos:
   - repo: https://github.com/psf/black-pre-commit-mirror
-    rev: 25.9.0
+    rev: 25.12.0
     hooks:
     - id: black
       exclude: \.py-tpl$
@@ -9,7 +9,7 @@ repos:
     hooks:
       - id: blacken-docs
         additional_dependencies:
-        - black==25.9.0
+        - black==25.12.0
         files: 'docs/.*\.txt$'
         args: ["--rst-literal-block"]
   - repo: https://github.com/PyCQA/isort
@@ -21,10 +21,10 @@ repos:
     hooks:
       - id: flake8
   - repo: https://github.com/pre-commit/mirrors-eslint
-    rev: v9.36.0
+    rev: v9.39.1
     hooks:
       - id: eslint
   - repo: https://github.com/zizmorcore/zizmor-pre-commit
-    rev: v1.16.3
+    rev: v1.19.0
     hooks:
     - id: zizmor

Thanks 👍

This should also be updated in the `package.json`.

Gotcha, see #20537.


Refs #24920 -- [NAME_REDACTED] system checks database dependent.

[NAME_REDACTED] for the suggestion.

ticket-24920

@@ -1715,40 +1715,46 @@ def __init__(

     def check(self, **kwargs):
         errors = super().check(**kwargs)
+        databases = kwargs.get("databases") or []

         digits_errors = [
-            *self._check_decimal_places(),
-            *self._check_max_digits(),
+            *self._check_decimal_places(databases),
+            *self._check_max_digits(databases),
         ]
         if not digits_errors:
             errors.extend(self._check_decimal_places_and_max_digits(**kwargs))
         else:
             errors.extend(digits_errors)
         return errors

-    def _check_decimal_places(self):
+    def _check_decimal_places(self, databases):
         if self.decimal_places is None:
-            if (
-                not connection.features.supports_no_precision_decimalfield
-                and "supports_no_precision_decimalfield"
-                not in self.model._meta.required_db_features
-            ):
-                return [
-                    checks.Error(
-                        "DecimalFields must define a 'decimal_places' attribute.",
-                        obj=self,
-                        id="fields.E130",
-                    )
-                ]
-            elif self.max_digits is not None:
-                return [
-                    checks.Error(
-                        "DecimalField’s max_digits and decimal_places must both "
-                        "be defined or both omitted.",
-                        obj=self,
-                        id="fields.E135",
-                    ),
-                ]
+            for db in databases:
+                if not router.allow_migrate_model(db, self.model):
+                    continue
+                connection = connections[db]
+
+                if (
+                    not connection.features.supports_no_precision_decimalfield
+                    and "supports_no_precision_decimalfield"
+                    not in self.model._meta.required_db_features
+                ):
+                    return [
+                        checks.Error(
+                            "DecimalFields must define a 'decimal_places' attribute.",
+                            obj=self,
+                            id="fields.E130",
+                        )
+                    ]
+                elif self.max_digits is not None:
+                    return [
+                        checks.Error(
+                            "DecimalField’s max_digits and decimal_places must both "
+                            "be defined or both omitted.",
+                            obj=self,
+                            id="fields.E135",
+                        ),
+                    ]
         else:
             try:
                 decimal_places = int(self.decimal_places)
@@ -1764,29 +1770,34 @@ def _check_decimal_places(self):
                 ]
         return []

-    def _check_max_digits(self):
+    def _check_max_digits(self, databases):
         if self.max_digits is None:
-            if (
-                not connection.features.supports_no_precision_decimalfield
-                and "supports_no_precision_decimalfield"
-                not in self.model._meta.required_db_features
-            ):
-                return [
-                    checks.Error(
-                        "DecimalFields must define a 'max_digits' attribute.",
-                        obj=self,
-                        id="fields.E132",
-                    )
-                ]
-            elif self.decimal_places is not None:
-                return [
-                    checks.Error(
-                        "DecimalField’s max_digits and decimal_places must both "
-                        "be defined or both omitted.",
-                        obj=self,
-                        id="fields.E135",
-                    ),
-                ]
+            for db in databases:
+                if not router.allow_migrate_model(db, self.model):
+                    continue
+                connection = connections[db]
+
+                if (
+                    not connection.features.supports_no_precision_decimalfield
+                    and "supports_no_precision_decimalfield"
+                    not in self.model._meta.required_db_features
+                ):
+                    return [
+                        checks.Error(
+                            "DecimalFields must define a 'max_digits' attribute.",
+                            obj=self,
+                            id="fields.E132",
+                        )
+                    ]
+                elif self.decimal_places is not None:
+                    return [
+                        checks.Error(
+                            "DecimalField’s max_digits and decimal_places must both "
+                            "be defined or both omitted.",
+                            obj=self,
+                            id="fields.E135",
+                        ),
+                    ]
         else:
             try:
                 max_digits = int(self.max_digits)

@@ -620,7 +620,7 @@ class Model(models.Model):
                     id="fields.E132",
                 ),
             ]
-        self.assertEqual(field.check(), expected)
+        self.assertEqual(field.check(databases=self.databases), expected)

     def test_both_attributes_omitted_required_db_features(self):
         class Model(models.Model):
@@ -639,7 +639,7 @@ class Model(models.Model):

         field = Model._meta.get_field("field")
         self.assertEqual(
-            field.check(),
+            field.check(databases=self.databases),
             [
                 Error(
                     "DecimalField’s max_digits and decimal_places must both "
@@ -657,7 +657,7 @@ class Model(models.Model):

         field = Model._meta.get_field("field")
         self.assertEqual(
-            field.check(),
+            field.check(databases=self.databases),
             [
                 Error(
                     "DecimalField’s max_digits and decimal_places must both "
@@ -674,7 +674,7 @@ class Model(models.Model):

         field = Model._meta.get_field("field")
         self.assertEqual(
-            field.check(),
+            field.check(databases=self.databases),
             [
                 Error(
                     "'decimal_places' must be a non-negative integer.",
@@ -695,7 +695,7 @@ class Model(models.Model):

         field = Model._meta.get_field("field")
         self.assertEqual(
-            field.check(),
+            field.check(databases=self.databases),
             [
                 Error(
                     "'decimal_places' must be a non-negative integer.",
@@ -731,7 +731,7 @@ class Model(models.Model):
             field = models.DecimalField(max_digits=10, decimal_places=10)

         field = Model._meta.get_field("field")
-        self.assertEqual(field.check(), [])
+        self.assertEqual(field.check(databases=self.databases), [])

 [USER_REDACTED]_apps("invalid_models_tests")

Any thoughts about moving this skip to `check`? I wonder if it might be worth adjusting `Model.check` to eventually skip delegation when `not router.allow_migrate_model(db, self.model)`.

I think you're missing `connection = connections[db]` here?

The second branch (when `decimal_places` is not `None`) should always run, that's why I put it here.

Yes, good catch :dart: Silly me :)


Fixed #36818 -- Ensured SQLite connection before accessing max_query_params.

Ensure that the SQLite database connection is established before accessing
DatabaseFeatures.max_query_params.

This fixes a regression where calling QuerySet.bulk_create() as the first
database operation after django.setup() could raise an AttributeError on
SQLite due to an uninitialized connection.

#### Trac ticket number
ticket-36818

#### Branch description
SQLite's DatabaseFeatures.max_query_params accessed the underlying DB-API
connection without ensuring it was initialized. This change calls
ensure_connection() before querying the SQLite variable limit, preventing
AttributeError when backend features are accessed prior to any SQL execution.

#### Checklist
- [x] This PR targets the `main` branch. <!-- Backports will be evaluated and done by mergers, when necessary. -->
- [x] The commit message is written in past tense, mentions the ticket number, and ends with a period.
- [x] I have checked the "Has patch" ticket flag in the Trac system.
- [x] I have added or updated relevant tests.
- [x] I have added or updated relevant docs, including release notes if applicable.
- [ ] I have attached screenshots in both light and dark modes for any UI changes.

@@ -157,6 +157,7 @@ def max_query_params(self):
         32766) or lowered per connection at run-time with
         setlimit(SQLITE_LIMIT_VARIABLE_NUMBER, N).
         """
+        self.connection.ensure_connection()
         return self.connection.connection.getlimit(sqlite3.SQLITE_LIMIT_VARIABLE_NUMBER)

     [USER_REDACTED]_property

@@ -20,3 +20,7 @@ Bugfixes
 * Fixed a bug where management command colorized help (introduced in
   Python 3.14) ignored the :option:`--no-color` option and the
   :envvar:`DJANGO_COLORS` setting (:ticket:`36376`).
+
+* Fixed a regression in Django 6.0 that caused
+  :meth:`~django.db.models.query.QuerySet.bulk_create` to crash
+  when introspecting the connection on SQLite (:ticket:`36818`).

@@ -1,3 +1,4 @@
+import copy
 import sqlite3
 from unittest import mock, skipUnless

@@ -29,3 +30,13 @@ def test_max_query_params_respects_variable_limit(self):
         finally:
             connection.connection.setlimit(limit_name, current_limit)
         self.assertEqual(connection.features.max_query_params, current_limit)
+
+    def test_max_query_params_without_established_connection(self):
+        new_connection = connection.copy()
+        new_connection.settings_dict = copy.deepcopy(connection.settings_dict)
+        self.assertIsNone(new_connection.connection)
+        try:
+            result = new_connection.features.max_query_params
+            self.assertIsInstance(result, int)
+        finally:
+            new_connection._close()

Thanks for PR [USER_REDACTED], really appreciated!

Feel free to add yourself to `AUTHORS` if you'd like to, I'll mark the patch as RFC.

Thanks 👍

Refer to 358fd21c47cdf7bda520ce73c5cfd82bba57827b and where the original test was added in `tests/backends/sqlite/test_features.py`.

The way this is currently defined the test is running on all backend when it should be SQLite specific.

https://github.com/[USER_REDACTED]ngo/django/commit/358fd21c47cdf7bda520ce73c5cfd82bba57827b#diff-bcc4842c09fdfbe9666b5d295f56846226e72148b015e0c19d4fc4ebd570b202R14-R22

```suggestion
* Fixed a regression in Django 6.0 that caused
  :meth:`~django.db.models.query.QuerySet.bulk_create` to crash
  when introspecting the connection on SQLite (:ticket:`36818`).
```

The problem is not only present if it's the first operation after `django.setup` as we also close connections between requests and the code was not ensuring the connection was opened either.

Updated the release notes as suggested.

I'm getting the following, when using an in-memory database:

```py
ResourceWarning: unclosed database in <sqlite3.Connection object at 0x10c710f40>
```
It goes away with:
```suggestion
            new_connection._close()
```

Thanks for the suggestion. I've updated the code to use `_close()` to ensure the underlying connection is properly closed in all environments.


Fixed #36787 -- Fixed crash in In lookups with mixed expressions and strings.

#### Trac ticket number
<!-- Replace XXXXX with the corresponding Trac ticket number, or delete the line and write "N/A" if this is a trivial PR. -->

ticket-36787

Thank you for providing the test code, [USER_REDACTED]

#### Checklist
- [x] This PR targets the `main` branch. <!-- Backports will be evaluated and done by mergers, when necessary. -->
- [x] The commit message is written in past tense, mentions the ticket number, and ends with a period.
- [x] I have checked the "Has patch" ticket flag in the Trac system.
- [x] I have added or updated relevant tests.
- [ ] I have added or updated relevant docs, including release notes if applicable.
- [ ] I have attached screenshots in both light and dark modes for any UI changes.

@@ -295,7 +295,7 @@ def get_prep_lookup(self):
                     (
                         value
                         if hasattr(value, "resolve_expression")
-                        else Value(value, self.lhs.output_field)
+                        else Value(value, getattr(self.lhs, "output_field", None))
                     )
                     for value in self.rhs
                 ]

@@ -1855,6 +1855,14 @@ def test_in_lookup_in_filter(self):
                     Season.objects.filter(In(F("year"), years)).order_by("pk"), seasons
                 )

+    def test_in_lookup_in_filter_text_field(self):
+        self.assertSequenceEqual(
+            Season.objects.filter(
+                In(F("nulled_text_field"), [F("nulled_text_field"), "special_value"])
+            ),
+            [self.s2],
+        )
+
     def test_filter_lookup_lhs(self):
         qs = Season.objects.annotate(before_20=LessThan(F("year"), 2000)).filter(
             before_20=LessThan(F("year"), 1900),

Thank you!

This test crashes on Oracle because blob fields (e.g. `NCLOB`) are not allowed in the `IN()` clause. As far as I'm aware `TextFields` are unnecessary here, see #20453.


Fixed #36376 -- Fixed color suppression in argparse help on Python 3.14+.

#### Trac ticket number
<!-- Replace XXXXX with the corresponding Trac ticket number, or delete the line and write "N/A" if this is a trivial PR. -->

ticket-36376

#### Branch description
This adds the argparse's colorized help feature for Django management commands when running on Python 3.14 or higher as suggested in the above mentioned ticket.

#### Checklist
- [x] This PR targets the `main` branch. <!-- Backports will be evaluated and done by mergers, when necessary. -->
- [x] The commit message is written in past tense, mentions the ticket number, and ends with a period.
- [x] I have checked the "Has patch" ticket flag in the Trac system.
- [x] I have added or updated relevant tests.
- [x] I have added or updated relevant docs, including release notes if applicable.
- [ ] I have attached screenshots in both light and dark modes for any UI changes.

@@ -58,8 +58,11 @@ def __init__(
     ):
         self.missing_args_message = missing_args_message
         self.called_from_command_line = called_from_command_line
-        if PY314 and not PY315:
-            kwargs.setdefault("suggest_on_error", True)
+        if PY314:
+            if not PY315:
+                kwargs.setdefault("suggest_on_error", True)
+            if os.environ.get("DJANGO_COLORS") == "nocolor" or "--no-color" in sys.argv:
+                kwargs.setdefault("color", False)
         super().__init__(**kwargs)

     def parse_args(self, args=None, namespace=None):

@@ -9,4 +9,6 @@ Django 5.2.10 fixes several bugs in 5.2.9.
 Bugfixes
 ========

-* ...
+* Fixed a bug where management command colorized help (introduced in
+  Python 3.14) ignored the :option:`--no-color` option and the
+  :envvar:`DJANGO_COLORS` setting (:ticket:`36376`).

@@ -16,3 +16,7 @@ Bugfixes
 * Fixed a regression in Django 6.0 that prevented changing the name of a
   :class:`~django.db.models.ManyToManyField` from taking effect when applying
   migrations (:ticket:`36800`).
+
+* Fixed a bug where management command colorized help (introduced in
+  Python 3.14) ignored the :option:`--no-color` option and the
+  :envvar:`DJANGO_COLORS` setting (:ticket:`36376`).

@@ -470,6 +470,34 @@ def test_suggest_on_error_explicit_false(self):
         )
         self.assertFalse(parser.suggest_on_error)

+    [USER_REDACTED].skipUnless(PY314, "Only relevant for Python 3.14+")
+    def test_color_enabled_by_default(self):
+        with mock.patch.dict(os.environ, {}, clear=True):
+            command = BaseCommand()
+            parser = command.create_parser("prog_name", "subcommand")
+            self.assertTrue(parser.color)
+
+    [USER_REDACTED].skipUnless(PY314, "Only relevant for Python 3.14+")
+    def test_color_disabled_with_django_colors_nocolor(self):
+        with mock.patch.dict(os.environ, {"DJANGO_COLORS": "nocolor"}):
+            command = BaseCommand()
+            parser = command.create_parser("prog_name", "subcommand")
+            self.assertFalse(parser.color)
+
+    [USER_REDACTED].skipUnless(PY314, "Only relevant for Python 3.14+")
+    def test_force_color_does_not_affect_argparse_color(self):
+        with mock.patch.dict(os.environ, {}, clear=True):
+            command = BaseCommand(force_color=True)
+            parser = command.create_parser("prog_name", "subcommand")
+            self.assertTrue(parser.color)
+
+    [USER_REDACTED].skipUnless(PY314, "Only relevant for Python 3.14+")
+    def test_no_color_flag_disables_color(self):
+        with mock.patch.object(sys, "argv", ["manage.py", "mycommand", "--no-color"]):
+            command = BaseCommand()
+            parser = command.create_parser("manage.py", "mycommand")
+            self.assertFalse(parser.color)
+

 class CommandRunTests(AdminScriptTestCase):
     """

Thanks [USER_REDACTED] 🎁

I left a comment on the ticket about how this has now morphed into a bugfix since the ticket was opened. Could you address the feedback and add a release note for 6.0.1.txt?

Thanks for the updates, this looks really good. Only some very minor feedback for you.

When the docstrings communicate the same information as the method names we chop them.

```suggestion
```
I don't think we need to bother; color is already True by [NAME_REDACTED]://github.com/python/cpython/pull/136809 (3.14)

This environment variable hasn't been set yet; we also need to check for an incoming ``--no-color`` argument instead. Please add a test.

I think we can remove this; this is more of a bugfix that we need to respect `--no-color` on python 3.14.

Point noted.

Great -- please duplicate this in 5.2.10.txt.

Very minor: I'd use "manage.py" and "mycommand" to match the mock.

I'd make all 4 assertions use either assertIs or not.

I think we should mention management commands up front, although it's fine to mention argparse in here.

Sorry I wasn't super clear -- we need a release note in both 5.2.10.txt and 6.0.1.txt.

Also, "management command" isn't a code literal, so please avoid backticks.

My bad , fixed it !

No worries, thank you!


Fixed #36786 -- Fixed XML serialization of None values in natural keys.

#### Trac ticket number
<!-- Replace XXXXX with the corresponding Trac ticket number, or delete the line and write "N/A" if this is a trivial PR. -->

[ticket-36786](https://code.djangoproject.com/ticket/36786)

#### Branch description

**AS-IS**:
- `None` values in natural keys were serialized as the string `"None"`, causing deserialization to fail.

**TO-BE**
- `None` values are serialized as `<None/>` elements, consistent with how regular fields handle `None`.

**Before**
```python
# Check for None first - but this matches <None/> inside <natural> tags too!
if node.getElementsByTagName("None"):
    return None
else:
    ...
    keys = node.getElementsByTagName("natural")
    if keys:
        field_value = [getInnerText(k).strip() for k in keys]
```

After

```python
natural_keys = node.getElementsByTagName("natural")
if node.getElementsByTagName("None") and not natural_keys:
    return None
else:
    ...
    if natural_keys:
        field_value = []
        for k in natural_keys:
            if k.getElementsByTagName("None"):
                field_value.append(None)
            else:
                field_value.append(getInnerText(k).strip())
```

#### Checklist
- [x] This PR targets the `main` branch. <!-- Backports will be evaluated and done by mergers, when necessary. -->
- [x] The commit message is written in past tense, mentions the ticket number, and ends with a period.
- [x] I have checked the "Has patch" ticket flag in the Trac system.
- [x] I have added or updated relevant tests.
- [ ] I have added or updated relevant docs, including release notes if applicable.
- [ ] I have attached screenshots in both light and dark modes for any UI changes.

@@ -134,7 +134,10 @@ def handle_fk_field(self, obj, field):
                 # Iterable natural keys are rolled out as subelements
                 for key_value in natural_key_value:
                     self.xml.startElement("natural", {})
-                    self.xml.characters(str(key_value))
+                    if key_value is None:
+                        self.xml.addQuickElement("None")
+                    else:
+                        self.xml.characters(str(key_value))
                     self.xml.endElement("natural")
             else:
                 self.xml.characters(str(related_att))
@@ -160,7 +163,10 @@ def handle_m2m(value):
                         self.xml.startElement("object", {})
                         for key_value in natural:
                             self.xml.startElement("natural", {})
-                            self.xml.characters(str(key_value))
+                            if key_value is None:
+                                self.xml.addQuickElement("None")
+                            else:
+                                self.xml.characters(str(key_value))
                             self.xml.endElement("natural")
                         self.xml.endElement("object")
                     else:
@@ -279,7 +285,11 @@ def _handle_object(self, node):
                 if value == base.DEFER_FIELD:
                     deferred_fields[field] = [
                         [
-                            getInnerText(nat_node).strip()
+                            (
+                                None
+                                if nat_node.getElementsByTagName("None")
+                                else getInnerText(nat_node).strip()
+                            )
                             for nat_node in obj_node.getElementsByTagName("natural")
                         ]
                         for obj_node in field_node.getElementsByTagName("object")
@@ -292,7 +302,11 @@ def _handle_object(self, node):
                 value = self._handle_fk_field_node(field_node, field)
                 if value == base.DEFER_FIELD:
                     deferred_fields[field] = [
-                        getInnerText(k).strip()
+                        (
+                            None
+                            if k.getElementsByTagName("None")
+                            else getInnerText(k).strip()
+                        )
                         for k in field_node.getElementsByTagName("natural")
                     ]
                 else:
@@ -317,16 +331,21 @@ def _handle_fk_field_node(self, node, field):
         Handle a <field> node for a ForeignKey
         """
         # Check if there is a child node named 'None', returning None if so.
-        if node.getElementsByTagName("None"):
+        natural_keys = node.getElementsByTagName("natural")
+        if node.getElementsByTagName("None") and not natural_keys:
             return None
         else:
             model = field.remote_field.model
             if hasattr(model._default_manager, "get_by_natural_key"):
-                keys = node.getElementsByTagName("natural")
-                if keys:
+                if natural_keys:
                     # If there are 'natural' subelements, it must be a natural
                     # key
-                    field_value = [getInnerText(k).strip() for k in keys]
+                    field_value = []
+                    for k in natural_keys:
+                        if k.getElementsByTagName("None"):
+                            field_value.append(None)
+                        else:
+                            field_value.append(getInnerText(k).strip())
                     try:
                         obj = model._default_manager.db_manager(
                             self.db
@@ -367,7 +386,12 @@ def m2m_convert(n):
                 if keys:
                     # If there are 'natural' subelements, it must be a natural
                     # key
-                    field_value = [getInnerText(k).strip() for k in keys]
+                    field_value = []
+                    for k in keys:
+                        if k.getElementsByTagName("None"):
+                            field_value.append(None)
+                        else:
+                            field_value.append(getInnerText(k).strip())
                     obj_pk = (
                         default_manager.db_manager(self.db)
                         .get_by_natural_key(*field_value)

@@ -91,3 +91,31 @@ class PostToOptOutSubclassUser(models.Model):
     subscribers = models.ManyToManyField(
         SubclassNaturalKeyOptOutUser, related_name="subscribed_posts", blank=True
     )
+
+
+class NaturalKeyWithNullableFieldManager(models.Manager):
+    def get_by_natural_key(self, name, optional_id):
+        return self.get(name=name, optional_id=optional_id)
+
+
+class NaturalKeyWithNullableField(models.Model):
+    name = models.CharField(max_length=100)
+    optional_id = models.CharField(max_length=100, null=True, blank=True)
+
+    objects = NaturalKeyWithNullableFieldManager()
+
+    class Meta:
+        unique_together = [["name", "optional_id"]]
+
+    def natural_key(self):
+        return (self.name, self.optional_id)
+
+
+class FKToNaturalKeyWithNullable(models.Model):
+    ref = models.ForeignKey(
+        NaturalKeyWithNullableField, on_delete=models.CASCADE, null=True
+    )
+    refs = models.ManyToManyField(
+        NaturalKeyWithNullableField, related_name="m2m_referrers"
+    )
+    data = models.CharField(max_length=100, default="")

@@ -6,8 +6,10 @@
     Child,
     FKAsPKNoNaturalKey,
     FKDataNaturalKey,
+    FKToNaturalKeyWithNullable,
     NaturalKeyAnchor,
     NaturalKeyThing,
+    NaturalKeyWithNullableField,
     NaturalPKWithDefault,
     PostToOptOutSubclassUser,
     SubclassNaturalKeyOptOutUser,
@@ -280,6 +282,60 @@ def natural_key_opt_out_test(self, format):
     )

+def nullable_natural_key_fk_test(self, format):
+    target_with_none = NaturalKeyWithNullableField.objects.create(
+        name="test_none",
+        optional_id=None,
+    )
+    target_with_value = NaturalKeyWithNullableField.objects.create(
+        name="test_value",
+        optional_id="some_id",
+    )
+    fk_to_none = FKToNaturalKeyWithNullable.objects.create(
+        ref=target_with_none,
+        data="points_to_none",
+    )
+    fk_to_value = FKToNaturalKeyWithNullable.objects.create(
+        ref=target_with_value,
+        data="points_to_value",
+    )
+    objects = [target_with_none, target_with_value, fk_to_none, fk_to_value]
+    serialized = serializers.serialize(
+        format,
+        objects,
+        use_natural_foreign_keys=True,
+        use_natural_primary_keys=True,
+    )
+    objs = list(serializers.deserialize(format, serialized))
+    self.assertEqual(objs[2].object.ref_id, target_with_none.pk)
+    self.assertEqual(objs[3].object.ref_id, target_with_value.pk)
+
+
+def nullable_natural_key_m2m_test(self, format):
+    target_with_none = NaturalKeyWithNullableField.objects.create(
+        name="test_none",
+        optional_id=None,
+    )
+    target_with_value = NaturalKeyWithNullableField.objects.create(
+        name="test_value",
+        optional_id="some_id",
+    )
+    m2m_obj = FKToNaturalKeyWithNullable.objects.create(data="m2m_test")
+    m2m_obj.refs.set([target_with_none, target_with_value])
+    objects = [target_with_none, target_with_value, m2m_obj]
+    serialized = serializers.serialize(
+        format,
+        objects,
+        use_natural_foreign_keys=True,
+        use_natural_primary_keys=True,
+    )
+    objs = list(serializers.deserialize(format, serialized))
+    self.assertCountEqual(
+        objs[2].m2m_data["refs"],
+        [target_with_none.pk, target_with_value.pk],
+    )
+
+
 # Dynamically register tests for each serializer
 register_tests(
     NaturalKeySerializerTests,
@@ -319,3 +375,13 @@ def natural_key_opt_out_test(self, format):
     "test_%s_natural_key_opt_out",
     natural_key_opt_out_test,
 )
+register_tests(
+    NaturalKeySerializerTests,
+    "test_%s_nullable_natural_key_fk",
+    nullable_natural_key_fk_test,
+)
+register_tests(
+    NaturalKeySerializerTests,
+    "test_%s_nullable_natural_key_m2m",
+    nullable_natural_key_m2m_test,
+)

Thanks -- especially appreciate the tests.

Main feedback at this point is to just prioritize a smaller diff over what we might otherwise do if rewriting this from scratch.

Thank you, looks great! 🎁

I'd rewrite this as a for loop.

For the sake of having a much smaller diff, I would suggest leaving most of this where it was and just fix the condition. (e.g. lift up `keys = ...` and then check `... and not keys: return None`, perhaps improving the variable name `keys`).

Your version has a terser if condition, but this code is touched so infrequently that I think we should prioritize a smaller diff and more understandable blames over more readable if conditions.

So my example of `UUIDField` on the ticket was just what I happened to be working with (and show a really nice validation problem when attempting the roundtrip). However, it's not very realistic as a member of a natural key (and doesn't work well with the yaml serializer). Could I ask you to try another field type?

Do we need the saving part, or can we just assert against the unsaved object and still capture the underlying issue?

You can remove these comments to blend in with the surrounding style.

Great models -- trying your tests against main shows all kinds of nasty failures 👍

Agreed, rewritten as a for loop.

I tried to simplify the logic, but ended up making a bigger diff than needed. Your suggestion is the better approach.
I've updated it.

Agreed. Changed to `CharField` it's more generalizable across all serializers and aligns with other natural key fields in the test suite.
e.g. (`NaturalKeyAnchor.data`, `NaturalKeyThing.key`).

From a serialization perspective, the responsibility is data transformation `save()` is a separate concern. I'll simplify the test. Thanks!

Thanks!


Fixed #36807 -- Fixed form field alignment under <fieldset> in the admin.

#### Trac ticket number
ticket-36807

#### Branch description
It isn't safe to set `display: flex` on `<fieldset>`, because on Safari, this interferes with `display: block` on child divs.

[NAME_REDACTED] for the report.

Regression in 4187da258fe212d494cb578a0bc2b52c4979ab95.

#### Checklist
- [x] This PR targets the `main` branch.
- [x] The commit message is written in past tense, mentions the ticket number, and ends with a period.
- [x] I have checked the "Has patch" ticket flag in the Trac system.
- [x] I have added or updated relevant tests.
- [x] I have added or updated relevant docs, including release notes if applicable.
- [x] I have attached screenshots in both light and dark modes for any UI changes.

#### Before

#### After (light)

#### After (dark)

@@ -102,7 +102,7 @@ fieldset .inline-heading,
 }

 .aligned legend {
-    float: left;
+    float: inline-start;
 }

 .aligned legend,
@@ -148,15 +148,11 @@ form .aligned ul {
 }

 form .aligned div.radiolist {
-    display: inline-block;
+    display: block;
     margin: 0;
     padding: 0;
 }

-form .aligned fieldset div.help {
-    margin-left: 0;
-}
-
 form .aligned p.help,
 form .aligned div.help {
     margin-top: 0;

@@ -560,6 +560,10 @@ input[type="submit"], button {
         margin-top: 5px;
     }

+    form .aligned fieldset div.flex-container {
+        display: unset;
+    }
+
     /* Related widget */

     .related-widget-wrapper {

@@ -40,7 +40,7 @@

   {{ form.usable_password.errors }}
-  <fieldset class="flex-container">{{ form.usable_password.legend_tag }} {{ form.usable_password }}</fieldset>
+  <fieldset>{{ form.usable_password.legend_tag }} {{ form.usable_password }}</fieldset>
   {% if form.usable_password.help_text %}

{{ form.usable_password.help_text|safe }}

@@ -11,7 +11,7 @@

             {% if line.fields|length == 1 %}{{ line.errors }}{% else %}
{% endif %}
             {% for field in line %}
-                {% if field.is_fieldset %}<fieldset class="flex-container"{% if field.field.help_text %} aria-describedby="{{ field.field.id_for_label }}_helptext"{% endif %}>{{ field.label_tag }}{% endif %}
+                {% if field.is_fieldset %}<fieldset{% if field.field.help_text %} aria-describedby="{{ field.field.id_for_label }}_helptext"{% endif %}>{{ field.label_tag }}{% endif %}

                     {% if not line.fields|length == 1 and not field.is_readonly %}{{ field.errors }}{% endif %}

@@ -24,3 +24,6 @@ Bugfixes
 * Fixed a regression in Django 6.0 that caused
   :meth:`~django.db.models.query.QuerySet.bulk_create` to crash
   when introspecting the connection on SQLite (:ticket:`36818`).
+
+* Fixed a visual regression in Django 6.0 for admin form fields grouped under a
+  ``<fieldset>`` in Safari (:ticket:`36807`).

@@ -1555,8 +1555,7 @@ def test_user_with_usable_password_change_password(self):
         # Usable password field.
         self.assertContains(
             response,
-            '<fieldset class="flex-container">'
-            "<legend>Password-based authentication:</legend>",
+            "<fieldset><legend>Password-based authentication:</legend>",
         )
         # Submit buttons
         self.assertContains(response, '<input type="submit" name="set-password"')

Thank you [USER_REDACTED], it looks good to me 🙂

While removing `flex`, I did notice a few additional layout issues that might be worth double checking:

- #### The help text for fieldset fields appears to be positioned differently compared to other fields.

It looks like a different `margin-left` is being applied for fieldsets.

- #### On mobile viewports, `fieldset` fields seem to disappear entirely.

I’m wondering if this could be related to them not inheriting the available width from the parent container.

- #### In RTL mode, the position of the legend in `fieldset` fields doesn’t seem to align with other fields.

This might be influenced by [NAME_REDACTED] `float: left`.

Thank you, Jacob 👍

I think we’re very close, just a few more things to tweak.

In addition to the review comments I left, it also looks like the `width` of the `datetime` fields is reduced on mobile screens.

<img width="814" height="128" alt="Screenshot 2025-12-19 at 11 29 03 AM" src="https://github.com/[USER_REDACTED]nts/assets/9a001921-8a27-4ddf-8f91-00e507108484" />

Is there a particular reason why `inline-block` is used on the `flex-container` under `fieldset` 🤔
Because of `inline-block`, the M2M fields end up being narrower, and when the screen is at desktop size and the field itself is wide, the label and field layout differs from the previous layout.

### m2m diff

## desktop screen diff

If there’s no specific reason for this, it seems like it would be okay to remove it  :)

I was trying to fix the mobile viewport issue, but good catch on the m2m impact. I'll try `display: unset`.

Actually, I just moved this selector into responsive.css

`unset` seems like a good solution.


Fixed #36804 -- Fixed admin system check crash for missing models.

…f [NAME_REDACTED] was an issue while raising the `NotRegistered` exception in get_model_admin when a string was passed to it instead of a model class. Fixed it by [NAME_REDACTED] `isinstance`. Added a test for this too.

#### Trac ticket number
<!-- Replace XXXXX with the corresponding Trac ticket number, or delete the line and write "N/A" if this is a trivial PR. -->

ticket-36804

#### Branch description
Handles a string passed to the `get_model_admin` method of the `AdminSite` class (in cases where the `to` field of a `ForeignKey` is specified using a model name instead of a model class.

#### Checklist
- [x] This PR targets the `main` branch. <!-- Backports will be evaluated and done by mergers, when necessary. -->
- [x] The commit message is written in past tense, mentions the ticket number, and ends with a period.
- [x] I have checked the "Has patch" ticket flag in the Trac system.
- [x] I have added or updated relevant tests.
- [ ] I have added or updated relevant docs, including release notes if applicable.
- [ ] I have attached screenshots in both light and dark modes for any UI changes.

@@ -236,14 +236,20 @@ def _check_autocomplete_fields_item(self, obj, field_name, label):
                     id="admin.E038",
                 )
             try:
+                if isinstance(field.remote_field.model, str):
+                    raise NotRegistered
                 related_admin = obj.admin_site.get_model_admin(field.remote_field.model)
             except NotRegistered:
+                # field.remote_field.model could be a string or a class.
+                remote_model = getattr(
+                    field.remote_field.model, "__name__", field.remote_field.model
+                )
                 return [
                     checks.Error(
                         'An admin for model "%s" has to be registered '
                         "to be referenced by %s.autocomplete_fields."
                         % (
-                            field.remote_field.model.__name__,
+                            remote_model,
                             type(obj).__name__,
                         ),
                         obj=obj.__class__,

@@ -1724,6 +1724,28 @@ class Admin(ModelAdmin):
             invalid_obj=Admin,
         )

+    [USER_REDACTED]_apps("modeladmin")
+    def test_autocomplete_e039_unresolved_model(self):
+        class UnresolvedForeignKeyModel(models.Model):
+            unresolved = models.ForeignKey("missing.Model", models.CASCADE)
+
+            class Meta:
+                app_label = "modeladmin"
+
+        class Admin(ModelAdmin):
+            autocomplete_fields = ("unresolved",)
+
+        self.assertIsInvalid(
+            Admin,
+            UnresolvedForeignKeyModel,
+            msg=(
+                'An admin for model "missing.Model" has to be registered '
+                "to be referenced by Admin.autocomplete_fields."
+            ),
+            id="admin.E039",
+            invalid_obj=Admin,
+        )
+
     def test_autocomplete_e040(self):
         class NoSearchFieldsAdmin(ModelAdmin):
             pass

Thank you ⭐

Thanks. I think we can leave the other example as is unless there is a way to reach it.

I left a possibility for you to consider to DRY this out, but we can also leave the explicitness of doing it twice if you prefer.

Thank you ⭐

Sorry, looking more closely I see that `get_model_admin` is documented to take a class, not a string. It might be better to solve this closer to the source of the erroring check, by checking `isinstance(..., str)` before calling `get_model_admin()`.

Makes sense. Will check and update soon.

The error came up when the `ForeignKey`'s `to` field was defined as a string, and the app was not registered. I have pushed a fix that checks the type before `get_model_admin` is called. I found that a similar call was made in `options.py`. Please let me know if it looks okay, will add regression tests if it does. Thanks!

```suggestion
                    raise NotRegistered
```

Then this could just be:
```
getattr(field.remote_field.model, "__name__", field.remote_field.model)
```
... with a nice comment. But if this is too complex for your taste, we can leave the exploded version.

I'm not sure this is realistic, wouldn't we have failed well before this?

Hmm ok. I'll remove the change in this PR and check separately.

Yeah this is much cleaner :) Will make this change.

Yep, looks great! I pushed minor formatting edits to reduce space. Be sure to do a hard reset back several commits and then pull my latest changes.

A regression test would be great, thanks.

Added the test


Fixed #36800 -- [NAME_REDACTED] renaming in BaseDatabaseSchemaEditor.alter_field().

Regression in f9a44cc0fac653f8e0c2ab1cdfb12b2cc5c63fc2, which removed column names from `ManyToManyField` instances but did not account for `attname` changes.

[NAME_REDACTED] alias [USER_REDACTED] for the report.

#### Trac ticket number
<!-- Replace XXXXX with the corresponding Trac ticket number, or delete the line and write "N/A" if this is a trivial PR. -->

ticket-36800

#### Checklist
- [x] This PR targets the `main` branch. <!-- Backports will be evaluated and done by mergers, when necessary. -->
- [x] The commit message is written in past tense, mentions the ticket number, and ends with a period.
- [x] I have checked the "Has patch" ticket flag in the Trac system.
- [x] I have added or updated relevant tests.
- [x] I have added or updated relevant docs, including release notes if applicable.

@@ -1690,7 +1690,7 @@ def _field_should_be_altered(self, old_field, new_field, ignore=None):
         _, old_path, old_args, old_kwargs = old_field.deconstruct()
         _, new_path, new_args, new_kwargs = new_field.deconstruct()
         # Don't alter when:
-        # - changing only a field name
+        # - changing only a field name (unless it's a many-to-many)
         # - changing an attribute that doesn't affect the schema
         # - changing an attribute in the provided set of ignored attributes
         # - adding only a db_column and the column name is not changed
@@ -1708,19 +1708,27 @@ def _field_should_be_altered(self, old_field, new_field, ignore=None):
         ):
             old_kwargs.pop("to", None)
             new_kwargs.pop("to", None)
-        # db_default can take many form but result in the same SQL.
+        # db_default can take many forms but result in the same SQL.
         if (
             old_kwargs.get("db_default")
             and new_kwargs.get("db_default")
             and self.db_default_sql(old_field) == self.db_default_sql(new_field)
         ):
             old_kwargs.pop("db_default")
             new_kwargs.pop("db_default")
-        return (
+        if (
             old_field.concrete
             and new_field.concrete
             and (self.quote_name(old_field.column) != self.quote_name(new_field.column))
-        ) or (old_path, old_args, old_kwargs) != (new_path, new_args, new_kwargs)
+        ):
+            return True
+        if (
+            old_field.many_to_many
+            and new_field.many_to_many
+            and old_field.name != new_field.name
+        ):
+            return True
+        return (old_path, old_args, old_kwargs) != (new_path, new_args, new_kwargs)

     def _field_should_be_indexed(self, model, field):
         return field.db_index and not field.unique

@@ -12,3 +12,7 @@ Bugfixes
 * Fixed a regression in Django 6.0 where :ttag:`querystring` mishandled
   multi-value :class:`~django.http.QueryDict` keys, both by [NAME_REDACTED] value and by [NAME_REDACTED] ``None`` values (:ticket:`36783`).
+
+* Fixed a regression in Django 6.0 that prevented changing the name of a
+  :class:`~django.db.models.ManyToManyField` from taking effect when applying
+  migrations (:ticket:`36800`).

@@ -2889,6 +2889,40 @@ def test_m2m_repoint_custom(self):
     def test_m2m_repoint_inherited(self):
         self._test_m2m_repoint(InheritedManyToManyField)

+    def test_m2m_rename(self):
+        class LocalBook(Model):
+            authors = ManyToManyField("schema.Author")
+
+            class Meta:
+                app_label = "schema"
+                apps = new_apps
+
+        self.local_models = [LocalBook]
+        with connection.schema_editor() as editor:
+            editor.create_model(Author)
+            editor.create_model(LocalBook)
+        old_field = LocalBook._meta.get_field("authors")
+        new_field = ManyToManyField("schema.Author")
+        new_field.contribute_to_class(LocalBook, "writers")
+        with connection.schema_editor() as editor:
+            editor.alter_field(LocalBook, old_field, new_field, strict=True)
+        # Ensure old M2M is gone.
+        with self.assertRaises(DatabaseError):
+            self.column_classes(
+                LocalBook._meta.get_field("authors").remote_field.through
+            )
+        if connection.features.supports_foreign_keys:
+            self.assertForeignKeyExists(
+                new_field.remote_field.through,
+                "author_id",
+                "schema_author",
+            )
+        new_through_table = new_field.remote_field.through._meta.db_table
+        self.assertIn("writers", new_through_table)
+        self.assertNotIn("authors", new_through_table)
+        # Remove the old field from meta for tearDown().
+        LocalBook._meta.local_many_to_many.remove(old_field)
+
     [USER_REDACTED]_apps("schema")
     def test_m2m_rename_field_in_target_model(self):
         class LocalTagM2MTest(Model):

I think we'll want to adjust the comment that says

```
# - changing only a field name
```

Otherwise testing at the schema level looks more appropriate than what I did in #20412

Perfect -- thanks [USER_REDACTED] and [USER_REDACTED] for the review.

Not sure why we're changing this.

Reverted 👍🏾. I'd changed it so it's easier to read.

[USER_REDACTED] After having a look at #20412, do you have an opinion about how to factor this? I think the multiple returns helps readability.


Refs #36769 -- [NAME_REDACTED] for unexpected nested tags in XML Deserializer.

#### Trac ticket number
ticket-36769

#### Branch description
[USER_REDACTED] observed that raising `SuspiciousOperation` would be preferable to silently discarding data from invalid fixtures in https://github.com/[USER_REDACTED]ngo/django/pull/20377#discussion_r2616789793.

#### Checklist
- [x] This PR targets the `main` branch. <!-- Backports will be evaluated and done by mergers, when necessary. -->
- [x] The commit message is written in past tense, mentions the ticket number, and ends with a period.
- [ ] I have checked the "Has patch" ticket flag in the Trac system.
- [x] I have added or updated relevant tests.
- [ ] I have added or updated relevant docs, including release notes if applicable.
- [ ] I have attached screenshots in both light and dark modes for any UI changes.

@@ -10,7 +10,7 @@

 from django.apps import apps
 from django.conf import settings
-from django.core.exceptions import ObjectDoesNotExist
+from django.core.exceptions import ObjectDoesNotExist, SuspiciousOperation
 from django.core.serializers import base
 from django.db import DEFAULT_DB_ALIAS, models
 from django.utils.xmlutils import SimplerXMLGenerator, UnserializableContentError
@@ -411,6 +411,8 @@ def m2m_convert(n):
         try:
             for c in node.getElementsByTagName("object"):
                 values.append(m2m_convert(c))
+        except SuspiciousOperation:
+            raise
         except Exception as e:
             if isinstance(e, ObjectDoesNotExist) and self.handle_forward_references:
                 return base.DEFER_FIELD
@@ -439,17 +441,15 @@ def _get_model_from_node(self, node, attr):
             )

+def check_element_type(element):
+    if element.childNodes:
+        raise SuspiciousOperation(f"Unexpected element: {element.tagName!r}")
+    return element.nodeType in (element.TEXT_NODE, element.CDATA_SECTION_NODE)
+
+
 def getInnerText(node):
-    """Get the inner text of a DOM node and any children one level deep."""
-    # inspired by
-    # https://mail.python.org/pipermail/xml-sig/2005-March/011022.html
     return "".join(
-        [
-            element.data
-            for child in node.childNodes
-            for element in (child, *child.childNodes)
-            if element.nodeType in (element.TEXT_NODE, element.CDATA_SECTION_NODE)
-        ]
+        [child.data for child in node.childNodes if check_element_type(child)]
     )

@@ -303,6 +303,10 @@ Serialization
   ``()``. This ensures primary keys are serialized when using
   :option:`dumpdata --natural-primary`.

+* The XML deserializer now raises
+  :exc:`~django.core.exceptions.SuspiciousOperation` when it encounters
+  unexpected nested tags.
+
 Signals
 ~~~~~~~

@@ -198,7 +198,8 @@ lowercase name of the model ("session") separated by a dot.

 Each field of the object is serialized as a ``<field>``-element sporting the
 fields "type" and "name". The text content of the element represents the value
-that should be stored.
+that should be stored. (If the element contains child tags,
+:exc:`~django.core.exceptions.SuspiciousOperation` is raised.)

 Foreign keys and other relational fields are treated a little bit differently:

@@ -237,6 +238,11 @@ This example links the given user with the permission models with PKs 46 and
     XHTML, XML and [NAME_REDACTED]
     <https://www.w3.org/International/questions/qa-controls>`_.

+.. versionchanged:: 6.1
+
+    :exc:`~django.core.exceptions.SuspiciousOperation` is raised when
+    unexpected nested tags are found.
+
 .. _serialization-formats-json:

 JSON

@@ -1,7 +1,7 @@
 <?xml version="1.0" encoding="utf-8"?>
 <django-objects version="1.0">
     <object pk="2" model="fixtures.visa">
-        <field type="CharField" name="person">
+        <field name="person" rel="ManyToOneRel" to="fixtures.person">
             <natural>[NAME_REDACTED]</natural>
         </field>
         <field to="auth.permission" name="permissions" rel="ManyToManyRel">
@@ -19,11 +19,11 @@
     </object>
     <object pk="3" model="fixtures.person">
         <field type="CharField" name="name">
-            <natural>Artist formerly known as "Prince"</natural>
+            Artist formerly known as "Prince"
         </field>
     </object>
     <object pk="3" model="fixtures.visa">
-        <field type="CharField" name="person">
+        <field name="person" rel="ManyToOneRel" to="fixtures.person">
             <natural>Artist formerly known as "Prince"</natural>
         </field>
         <field to="auth.permission" name="permissions" rel="ManyToManyRel">

@@ -3,7 +3,8 @@

     <object pk="1" model="fixtures.person">
         <field type="CharField" name="name">
-            <natural>Django pony</natural>
+
+            Django pony
         </field>
     </object>

@@ -0,0 +1,14 @@
+<?xml version="1.0" encoding="utf-8"?>
+<django-objects version="1.0">
+
+    <object pk="1" model="fixtures.book">
+        <field type="CharField" name="name">Music for all ages</field>
+        <field to="fixtures.person" name="authors" rel="ManyToManyRel">
+            <object>
+
+                <natural>Artist formerly known as Prince</natural>
+            </object>
+        </field>
+    </object>
+
+</django-objects>

@@ -10,6 +10,7 @@
 from django.apps import apps
 from django.contrib.sites.models import Site
 from django.core import management
+from django.core.exceptions import SuspiciousOperation
 from django.core.files.temp import NamedTemporaryFile
 from django.core.management import CommandError
 from django.core.management.commands.dumpdata import ProxyModelWarning
@@ -23,7 +24,6 @@
     CircularA,
     CircularB,
     NaturalKeyThing,
-    Person,
     PrimaryKeyUUIDModel,
     ProxySpy,
     Spy,
@@ -522,12 +522,13 @@ def test_loading_and_dumping(self):
         )

     def test_deeply_nested_elements(self):
-        """Text inside deeply-nested tags is skipped."""
-        management.call_command(
-            "loaddata", "invalid_deeply_nested_elements.xml", verbosity=0
-        )
-        person = Person.objects.get(pk=1)
-        self.assertEqual(person.name, "Django")  # not "Django pony"
+        """Text inside deeply-nested tags raises SuspiciousOperation."""
+        for file in [
+            "invalid_deeply_nested_elements.xml",
+            "invalid_deeply_nested_elements_natural_key.xml",
+        ]:
+            with self.subTest(file=file), self.assertRaises(SuspiciousOperation):
+                management.call_command("loaddata", file, verbosity=0)

     def test_dumpdata_with_excludes(self):
         # Load fixture1 which has a site, two articles, and a category

@@ -1,15 +1,14 @@
 import json
-import time
+import textwrap
 import unittest

+from django.core.exceptions import SuspiciousOperation
 from django.core.serializers.base import DeserializationError, DeserializedObject
 from django.core.serializers.json import Deserializer as JsonDeserializer
 from django.core.serializers.jsonl import Deserializer as JsonlDeserializer
 from django.core.serializers.python import Deserializer
 from django.core.serializers.xml_serializer import Deserializer as XMLDeserializer
-from django.db import models
 from django.test import SimpleTestCase
-from django.test.utils import garbage_collect

 from .models import Author

@@ -138,52 +137,23 @@ def test_yaml_bytes_input(self):
         self.assertEqual(first_item.object, self.jane)
         self.assertEqual(second_item.object, self.joe)

-    def test_crafted_xml_performance(self):
-        """The time to process invalid inputs is not quadratic."""
-
-        def build_crafted_xml(depth, leaf_text_len):
-            nested_open = "<nested>" * depth
-            nested_close = "</nested>" * depth
-            leaf = "x" * leaf_text_len
-            field_content = f"{nested_open}{leaf}{nested_close}"
-            return f"""
-                <django-objects version="1.0">
-                   <object model="contenttypes.contenttype" pk="1">
-                      <field name="app_label">{field_content}</field>
-                      <field name="model">m</field>
-                   </object>
-                </django-objects>
-            """
-
-        def deserialize(crafted_xml):
-            iterator = XMLDeserializer(crafted_xml)
-            garbage_collect()
-
-            start_time = time.perf_counter()
-            result = list(iterator)
-            end_time = time.perf_counter()
-
-            self.assertEqual(len(result), 1)
-            self.assertIsInstance(result[0].object, models.Model)
-            return end_time - start_time
-
-        def assertFactor(label, params, factor=2):
-            factors = []
-            prev_time = None
-            for depth, length in params:
-                crafted_xml = build_crafted_xml(depth, length)
-                elapsed = deserialize(crafted_xml)
-                if prev_time is not None:
-                    factors.append(elapsed / prev_time)
-                prev_time = elapsed
-
-            with self.subTest(label):
-                # Assert based on the average factor to reduce test flakiness.
-                self.assertLessEqual(sum(factors) / len(factors), factor)
-
-        assertFactor(
-            "varying depth, varying length",
-            [(50, 2000), (100, 4000), (200, 8000), (400, 16000), (800, 32000)],
-            2,
+    def test_crafted_xml_rejected(self):
+        depth = 100
+        leaf_text_len = 1000
+        nested_open = "<nested>" * depth
+        nested_close = "</nested>" * depth
+        leaf = "x" * leaf_text_len
+        field_content = f"{nested_open}{leaf}{nested_close}"
+        crafted_xml = textwrap.dedent(
+            f"""
+        <django-objects version="1.0">
+            <object model="contenttypes.contenttype" pk="1">
+                <field name="app_label">{field_content}</field>
+                <field name="model">m</field>
+            </object>
+        </django-objects>"""
         )
-        assertFactor("constant depth, varying length", [(100, 1), (100, 1000)], 2)
+
+        msg = "Unexpected element: 'nested'"
+        with self.assertRaisesMessage(SuspiciousOperation, msg):
+            list(XMLDeserializer(crafted_xml))

Hi,

This seems not-yet-ready to me. Notes inside, some of taste and preference, but some of apparent correctness.

Thank you! 🌟

LGTM (up to [USER_REDACTED] 's test concerns). I see some issues, but they're out of scope for this PR.

This looks great, thank you [USER_REDACTED]! Added a minor ask, but I think this is good to go.

I moved the `<natural>` key example to another file.

There might be another one of these to filter out and raise, e.g. in `save_deferred_fields()`.

Undecided between "Node" vs "Tag".

I'd be more general, `SuspiciousXML` or even `SuspiciousSerialization`.

But note that, currently, AFAICT, all `SuspiciousOperation` child classes in the Django source are defined in `exceptions.py` files -- and except for those in `contrib` apps, all are in `django.core.exceptions`. I suspect we could do without a child class at all.

Why give special treatment to `natural` nodes inside fields which aren't relation fields?

This is a good idea IMO, but, regardless of whether we need a specific exception subclass for this fix, I'd go with
```suggestion
        except SuspiciousOperation:
```

This doesn't look equivalent... the original took children and grandchildren, this drops grandchildren.

`build_crafted_xml` seems redundant as a function if it only gets called once. I'd simplify it away.

Right, I should have left notes about this to aid the review. This change is possible _only_ in virtue of the other change you noticed, where I'm specializing `<natural>` nodes.

That's the only location where we need to drill two levels deep, e.g. from:

```
<field>
    <natural> <!-- child -->
        some content <!-- grandchild text node -->
```

Grandchild text nodes don't need to be visited if I instead call the helper on `<natural>` nodes. That dramatically simplifies checking for unexpected children.

Elsewhere, `<field>` nodes are interrogated when we _don't_ expect `<natural>` serialization, e.g.

```
<field>
    some content <!-- child text node -->
```

Thanks for the review. Sorry I didn't share more of my findings. I think throughout the serializer natural nodes are specialized. The ones that are relations, in the branches just before this, are specialized inside `_handle_m2m_field_node`. Unless I'm missing your meaning.

I checked again and couldn't find any other places to update.

See the subtest iteration in `test_deeply_nested_elements` testing one example with natural keys and one without.

[USER_REDACTED] I think the key point here is that `<natural>` is not limited to FK or M2M values. It can also appear on direct fields when those fields participate in the object's own natural key (e.g. fixtures produced with `dumpdata --natural-primary`, as Jacob patiently explained to me when we chatted). In that sense, handling `<natural>` outside the relation branches is expected, not special-casing.

With that understanding, the change looks good to me. The only remaining concern is test coverage: reverting this change (but leaving `getInnerText` as modified in this commit) would not currently cause any serializer tests to fail, so it would be good to see a few targeted tests added to lock this behavior in.

I guess `<natural>` is the exception here? as shown in https://github.com/[USER_REDACTED]ngo/django/blob/main/tests/fixtures/fixtures/fixture9.xml

[USER_REDACTED] [USER_REDACTED] Thanks for pushing me to test my understanding here. I tried dumping fixture9, and noticed it was different from the `fixture9.xml` in the repo. Ouch.

I think this all boils down to an invalid fixture. Upon correcting the fixture, I was able to remove this specialization.

> In that sense, handling <natural> outside the relation branches is expected, not special-casing.

I managed to talk [USER_REDACTED] into this when we we chatted, but it was wrong -- I think I was not prepared for the fixture to turn out to be invalid in *two* ways, not only whether or not `<natural>` should be present, but whether the definitions of the fields were to be trusted (missing `rel=ManyToOne...`).

I wonder if it was just a copy-paste mistake in 35cc439.

Yeah. I'm not sure whether to detour into that here. Natural keys are explained below, I'm very hesitant to detour into it beforehand.

I've looked at that documentation, and I agree with [USER_REDACTED] -- I see no need to get into natural keys at this point.

That said, most of the examples there are given with JSON (understandably), and with JSON, you just use the value of the natural keys instead of the value of the PK -- there's no `"natural"` key or anything of the sort, to parallel the `<natural>` element. The docs say nothing of it, and they probably should. This should probably be treated as a separate issue.

(on second thought, another thing to treat as a separate issue is -- do we really want to support XML serialization going forward?)

Could we perhaps add a message to this exception, since we are re-raising it in `_handle_m2m_field_node`? To aid debugging/understanding the error.


Fixed #36781 -- Added Granian to deployment documentation.

#### Trac ticket number
<!-- Replace XXXXX with the corresponding Trac ticket number, or delete the line and write "N/A" if this is a trivial PR. -->

ticket-36781

#### Branch description
Granian is a Rust-based high-performance HTTP server that supports both ASGI and WSGI interfaces. This patch adds deployment documentation following the
  same structure as existing server docs (Daphne, Uvicorn, Hypercorn, Gunicorn).

- https://forum.djangoproject.com/t/add-granian-to-how-to-deploy-docs-asgi-wsgi-support/43386

#### Checklist
- [x] This PR targets the `main` branch. <!-- Backports will be evaluated and done by mergers, when necessary. -->
- [x] The commit message is written in past tense, mentions the ticket number, and ends with a period.
- [x] I have checked the "Has patch" ticket flag in the Trac system.
- [ ] I have added or updated relevant tests.
- [x] I have added or updated relevant docs, including release notes if applicable.
- [ ] I have attached screenshots in both light and dark modes for any UI changes.

@@ -0,0 +1,37 @@
+==============================
+How to use Django with Granian
+==============================
+
+Granian_ is a Rust-based high-performance HTTP server for Python applications,
+supporting ASGI and WSGI interfaces.
+
+.. _Granian: https://github.com/emmett-framework/granian
+
+[NAME_REDACTED]
+==================
+
+You can install Granian with ``pip``:
+
+.. code-block:: shell
+
+    python -m pip install granian
+
+[NAME_REDACTED] in Granian
+=========================
+
+When Granian is installed, a ``granian`` command is available which runs ASGI
+applications. Granian needs to be called with the location of a module
+containing an ASGI application object, followed by [NAME_REDACTED]
+called (separated by a colon), using the ``--interface asgi`` option.
+
+For a typical Django project, invoking Granian would look like:
+
+.. code-block:: shell
+
+    granian myproject.asgi:application --interface asgi
+
+This will start one process listening on ``127.0.0.1:8000``. It requires that
+your project be on the Python path; to ensure that, run this command from the
+same directory as your ``manage.py`` file.
+
+For more advanced usage, please read the `Granian documentation <Granian_>`_.

@@ -17,6 +17,7 @@ Django includes getting-started documentation for the following ASGI servers:
    :maxdepth: 1

    daphne
+   granian
    hypercorn
    uvicorn

@@ -0,0 +1,37 @@
+==============================
+How to use Django with Granian
+==============================
+
+Granian_ is a Rust-based high-performance HTTP server for Python applications,
+supporting ASGI and WSGI interfaces.
+
+.. _Granian: https://github.com/emmett-framework/granian
+
+[NAME_REDACTED]
+==================
+
+You can install Granian with ``pip``:
+
+.. code-block:: shell
+
+    python -m pip install granian
+
+[NAME_REDACTED] in Granian
+=========================
+
+When Granian is installed, a ``granian`` command is available which runs WSGI
+applications. Granian needs to be called with the location of a module
+containing a WSGI application object named ``application``, using the
+``--interface wsgi`` option.
+
+For a typical Django project, invoking Granian would look like:
+
+.. code-block:: shell
+
+    granian myproject.wsgi:application --interface wsgi
+
+This will start one process listening on ``127.0.0.1:8000``. It requires that
+your project be on the Python path; to ensure that, run this command from the
+same directory as your ``manage.py`` file.
+
+For more advanced usage, please read the `Granian documentation <Granian_>`_.

@@ -16,6 +16,7 @@ Django includes getting-started documentation for the following WSGI servers:
 .. toctree::
    :maxdepth: 1

+   granian
    gunicorn
    uwsgi
    modwsgi

@@ -190,6 +190,7 @@ gis
 [NAME_REDACTED]
+Granian
 gunicorn
 GZip
 gzipped

Thanks [USER_REDACTED] ⭐

This invites the question "what is RSGI, and does Django support it"? I'd avoid mentioning it.

Thank you for the review, Jacob. I agree that it could be ambiguous, so I’ve revised it.

I'd do the same change in the other file.

a bit nitpicking, but since in other server docs we format it like:
`uvicorn myproject.asgi:application`, it might be cleaner to format this the same way

```suggestion
    granian myproject.asgi:application --interface asgi
```

```suggestion
your project be on the Python path; to ensure that, run this command from the
```

```suggestion
    granian myproject.wsgi:application --interface wsgi
```


…ToManyField target model changes.

#### Trac ticket number
<!-- Replace XXXXX with the corresponding Trac ticket number, or delete the line and write "N/A" if this is a trivial PR. -->

ticket-[36791](https://code.djangoproject.com/ticket/36791)

#### Branch description
Provide a concise overview of the issue or rationale behind the proposed changes.

#### Checklist
- [x] This PR targets the `main` branch. <!-- Backports will be evaluated and done by mergers, when necessary. -->
- [x] The commit message is written in past tense, mentions the ticket number, and ends with a period.
- [x] I have checked the "Has patch" ticket flag in the Trac system.
- [x] I have added or updated relevant tests.
- [x] I have added or updated relevant docs, including release notes if applicable.
- [ ] I have attached screenshots in both light and dark modes for any UI changes.

@@ -1339,7 +1339,10 @@ def generate_altered_fields(self):
             if old_field_dec != new_field_dec and old_field_name == field_name:
                 both_m2m = old_field.many_to_many and new_field.many_to_many
                 neither_m2m = not old_field.many_to_many and not new_field.many_to_many
-                if both_m2m or neither_m2m:
+                target_changed = (
+                    both_m2m and new_field_dec[2]["to"] != old_field_dec[2]["to"]
+                )
+                if (both_m2m or neither_m2m) and not target_changed:
                     # Either both fields are m2m or neither is
                     preserve_default = True
                     if (

@@ -741,6 +741,13 @@ class AutodetectorTests(BaseAutodetectorTests):
             ("publishers", models.ManyToManyField("testapp.Publisher", blank=True)),
         ],
     )
+    other_publisher = ModelState(
+        "testapp",
+        "OtherPublisher",
+        [
+            ("id", models.AutoField(primary_key=True)),
+        ],
+    )
     author_with_m2m_through = ModelState(
         "testapp",
         "Author",
@@ -4557,6 +4564,46 @@ def test_many_to_many_changed_to_concrete_field(self):
         )
         self.assertOperationFieldAttributes(changes, "testapp", 0, 2, max_length=100)

+    def test_m2m_target_change_generates_remove_and_add(self):
+        before = [
+            self.publisher,
+            self.other_publisher,
+            self.author_with_m2m,  # m2m to self.publisher.
+        ]
+
+        after = [
+            self.publisher,
+            self.other_publisher,
+            ModelState(
+                "testapp",
+                "Author",
+                [
+                    ("id", models.AutoField(primary_key=True)),
+                    # Repoint m2m to self.other_publisher.
+                    ("publishers", models.ManyToManyField("testapp.OtherPublisher")),
+                ],
+            ),
+        ]
+        changes = self.get_changes(before, after)
+        self.assertNumberMigrations(changes, "testapp", 1)
+        self.assertOperationTypes(changes, "testapp", 0, ["RemoveField", "AddField"])
+        self.assertOperationAttributes(
+            changes,
+            "testapp",
+            0,
+            0,
+            name="publishers",
+            model_name="author",
+        )
+        self.assertOperationAttributes(
+            changes,
+            "testapp",
+            0,
+            1,
+            name="publishers",
+            model_name="author",
+        )
+
     def test_non_circular_foreignkey_dependency_removal(self):
         """
         If two models with a ForeignKey from one to the other are removed at

Thanks [USER_REDACTED]. Looks great 🏆

I added some suggestions

Thanks for the updates 👍🏾

You will need to squash the commits and fix the title, e.g. `Fixed #36791 -- [NAME_REDACTED] recreate through table when m2m target model changes.` and add Jacob as co-author

Small detail: you forgot to clean up your commit message body to remove the old messages 😸 But otherwise looks great.

Thanks 🏆

Thanks for the updates 👍🏾

We only need to add a variable for something we need to use more than once, in this case the other publisher. I have added suggestions doing that, and replacing `Magazine` with `OtherPublisher` (just for semantics).

I'd also move the test to below `test_many_to_many_changed_to_concrete_field` so it is grouped with other m2m-related tests.

Thanks 🚀

Thanks [USER_REDACTED] and thanks [USER_REDACTED] for the review!

use `assertNumberMigrations()` (see surrounding tests)

use `assertOperationTypes()`

use `assertOperationAttributes()`

Nit: I'd remove the blank lines; the file is already too long as it is
```suggestion
        changes = self.get_changes(before, after)
        self.assertNumberMigrations(changes, "testapp", 1)
        self.assertOperationTypes(changes, "testapp", 0, ["RemoveField", "AddField"])
        self.assertOperationAttributes(
            changes,
            "testapp",
            0,
            0,
            name="readings",
            model_name="author",
        )
        self.assertOperationAttributes(
```

I've just discovered these model states or similar are predefined, e.g. `self.author_with_m2m`. Can you switch to those here and, where appropriate, in `after` as well.

```suggestion
        before = [
            self.publisher,
            self.other_publisher,
            self.author_with_m2m,  # m2m to self.publisher.
```

I think we only need a variable for something we need to use more than once.
```suggestion
    other_publisher = ModelState(
        "testapp",
        "OtherPublisher",
        [
            ("id", models.AutoField(primary_key=True)),
        ],
    )
```

```suggestion
        after = [
            self.publisher,
            self.other_publisher,
            ModelState(
                "testapp",
                "Author",
                [
                    ("id", models.AutoField(primary_key=True)),
                    # Repoint m2m to self.other_publisher.
                    ("publishers", models.ManyToManyField("testapp.OtherPublisher")),
                ],
            ),
        ]
```

We missed this outdated comment

And also if I could bother you to move the test, as suggested [here](https://github.com/[USER_REDACTED]ngo/django/pull/20395#pullrequestreview-3592776256).

Otherwise this should be good to go.


Fixed #36701 -- Fixed memory leak in ModelState.

#### Trac ticket number

ticket-36701

#### Branch description
This PR addresses ticket #36701 regarding memory leaks caused by [NAME_REDACTED] `ModelState`

I implemented a `__del__` method on `ModelState` to explicitly clear `self.fields_cache`.

I added a regression test (`tests/model_state/test_memory_leak.py`) that:
1. Creates a `OneToOneField` cycle.
2. Deletes strong references.
3. Forces a `gc.collect()`.
4. Asserts that the objects are successfully reclaimed (using `weakref`).

I also verified this locally with a reproduction script generating 50,000 cyclic objects. Without the fix, memory usage balloons. With the fix, `gc.collect()` successfully reclaims the objects.
#### Checklist
- [ ] This PR targets the `main` branch. <!-- Backports will be evaluated and done by mergers, when necessary. -->
- [ ] The commit message is written in past tense, mentions the ticket number, and ends with a period.
- [ ] I have checked the "Has patch" ticket flag in the Trac system.
- [ ] I have added or updated relevant tests.
- [ ] I have added or updated relevant docs, including release notes if applicable.
- [ ] I have attached screenshots in both light and dark modes for any UI changes.

@@ -494,6 +494,9 @@ def __getstate__(self):
         state.pop("peers", None)
         return state

+    def __del__(self):
+        self.fields_cache.clear()
+

 class Model(AltersData, metaclass=ModelBase):
     def __init__(self, *args, **kwargs):

@@ -45,6 +45,10 @@ def __str__(self):
         return self.name

+class WorkerProfile(models.Model):
+    worker = models.OneToOneField(Worker, on_delete=models.CASCADE)
+
+
 class NonAutoPK(models.Model):
     name = models.CharField(max_length=10, primary_key=True)

@@ -1,7 +1,33 @@
+import gc
+
 from django.db.models.base import ModelState, ModelStateFieldsCacheDescriptor
 from django.test import SimpleTestCase
+from django.test.utils import garbage_collect
+
+from .models import Worker, WorkerProfile

 class ModelStateTests(SimpleTestCase):
     def test_fields_cache_descriptor(self):
         self.assertIsInstance(ModelState.fields_cache, ModelStateFieldsCacheDescriptor)
+
+    def test_one_to_one_field_cycle_collection(self):
+        self.addCleanup(gc.set_debug, gc.get_debug())
+        gc.set_debug(gc.DEBUG_SAVEALL)
+
+        def clear_garbage():
+            del gc.garbage[:]
+
+        self.addCleanup(clear_garbage)
+
+        worker = Worker()
+        profile = WorkerProfile(worker=worker)
+        worker_id = id(worker)
+
+        del worker
+        del profile
+
+        garbage_collect()
+
+        leaked = [obj for obj in gc.garbage if id(obj) == worker_id]
+        self.assertEqual(leaked, [])

Thanks 👍

Thanks for the updates 👍

Thanks, I pushed small edits to tests to reuse a model.

One last thing to polish.

Thanks 👍

You can remove this in favor of a more descriptive method title mentioning one_to_one (or o2o) fields.

This test doesn't fail when I revert your changes. You might be able to draw inspiration from this:

https://github.com/[USER_REDACTED]ngo/django/blob/334308efae8e0c7b1523d5583af32b674a098eba/tests/select_related/tests.py#L63-L72

Please move this test to tests/model_regress/test_state.py

I am really sorry, I saw the test passing and got excited and forgot to check for the main scenario I was writing the test.
I have written the test again and verified it is failing now before the fix and passing after the fix

Why do we need the except?

Please use the `garbage_collect()` helper from `django.test.utils`.

Why do we need this?

```suggestion
```
Thanks for the method name change. Now we can lose the docstring (we avoid "Ensure ..." preambles anyway.)

After this test I get this warning:
```py
Exception ignored in GC shutdown:
ResourceWarning: gc: 7995 uncollectable objects at shutdown; use gc.set_debug(gc.DEBUG_UNCOLLECTABLE) to list them
```

My question was getting at, under what conditions would `_fields_cache` be missing?

This test is testing the implementation instead of the underlying issue. It would be better to polish the test you had before.

Thanks for explaining -- in that case these lines are reachable, so we need a test to cover them.

Again, thanks very much for explaining. Now we need to register this as a cleanup earlier (right after garbage_collect, with `self.addCleanup`) because if the assertion fails for some reason I get the warning again.

This is still not covered.

Where is `_fields_cache` assigned?

[USER_REDACTED]

Regarding `_fields_cache` assignment: That was an error in my test code. The internal storage key is actually `fields_cache` . I have corrected the test to check state.`__dict__` for the correct key `("fields_cache")`.

Reliance on `del state` alone was causing coverage gaps. I have updated the test to explicitly call state.`__del__()`. This forces the execution of the except AttributeError: pass block, ensuring 100% coverage for the teardown logic.

The except is still not covered. Also, please fetch the changes I pushed to your branch; you've force-pushed over them. TIP: you can use `--force-with-lease` when force pushing to ensure you get an error if there are missing changes to fetch first.

```suggestion
```

Sorry, I guess this whole test method can go now, now that we're not trying to exercise some separate code path.

ok


Fixed #36769 -- Avoided visiting deeply nested nodes in XML deserializer.

#### Trac ticket number
<!-- Replace XXXXX with the corresponding Trac ticket number, or delete the line and write "N/A" if this is a trivial PR. -->

ticket-36769

#### Branch description
This change adds a limit to how deep getInnerTextList() will read XML tags.
Django fixtures only need one level of text (for example: <field><natural>text</natural>).
Deeper levels are not used and can cause performance problems if the XML is very nested.

#### Checklist
- [x] This PR targets the `main` branch. <!-- Backports will be evaluated and done by mergers, when necessary. -->
- [ ] The commit message is written in past tense, mentions the ticket number, and ends with a period.
- [ ] I have checked the "Has patch" ticket flag in the Trac system.
- [ ] I have added or updated relevant tests.
- [ ] I have added or updated relevant docs, including release notes if applicable.
- [ ] I have attached screenshots in both light and dark modes for any UI changes.

@@ -416,27 +416,17 @@ def _get_model_from_node(self, node, attr):

 def getInnerText(node):
-    """Get all the inner text of a DOM node (recursively)."""
-    inner_text_list = getInnerTextList(node)
-    return "".join(inner_text_list)
-
-
-def getInnerTextList(node):
-    """Return a list of the inner texts of a DOM node (recursively)."""
+    """Get the inner text of a DOM node and any children one level deep."""
     # inspired by
     # https://mail.python.org/pipermail/xml-sig/2005-March/011022.html
-    result = []
-    for child in node.childNodes:
-        if (
-            child.nodeType == child.TEXT_NODE
-            or child.nodeType == child.CDATA_SECTION_NODE
-        ):
-            result.append(child.data)
-        elif child.nodeType == child.ELEMENT_NODE:
-            result.extend(getInnerTextList(child))
-        else:
-            pass
-    return result
+    return "".join(
+        [
+            element.data
+            for child in node.childNodes
+            for element in (child, *child.childNodes)
+            if element.nodeType in (element.TEXT_NODE, element.CDATA_SECTION_NODE)
+        ]
+    )

 # Below code based on [NAME_REDACTED]' defusedxml

@@ -0,0 +1,10 @@
+<?xml version="1.0" encoding="utf-8"?>
+<django-objects version="1.0">
+
+    <object pk="1" model="fixtures.person">
+        <field type="CharField" name="name">
+            <natural>Django pony</natural>
+        </field>
+    </object>
+
+</django-objects>

@@ -23,6 +23,7 @@
     CircularA,
     CircularB,
     NaturalKeyThing,
+    Person,
     PrimaryKeyUUIDModel,
     ProxySpy,
     Spy,
@@ -520,6 +521,14 @@ def test_loading_and_dumping(self):
             natural_foreign_keys=True,
         )

+    def test_deeply_nested_elements(self):
+        """Text inside deeply-nested tags is skipped."""
+        management.call_command(
+            "loaddata", "invalid_deeply_nested_elements.xml", verbosity=0
+        )
+        person = Person.objects.get(pk=1)
+        self.assertEqual(person.name, "Django")  # not "Django pony"
+
     def test_dumpdata_with_excludes(self):
         # Load fixture1 which has a site, two articles, and a category
         Site.objects.all().delete()

Thanks 👍

Can you add a test to show the behavior change? I'm imagining a new file in `tests/fixtures/fixtures` like `invalid_deeply_nested_elements.xml` showing that text inside unescaped markup like `<natural>Django <em>pony</em></natural>` now just extracts "Django " rather than "Django pony".

Then, it will be more easy to tell if we even need the max_depth at all. My preliminary testing is showing that we don't need to recurse at _all_, since we call `getInnerText` directly on `<natural>` elements.

Thanks ⭐

Sorry to come in late... one comment of substance below.

Limit max recursion depth to 1 level because fixtures only need text one level deep

This docstring needs an edit, as we're no longer getting "all" inner text.

changed the doc string wording here

Thanks, but `getInnerText()` is still not correct.

my bad, i missed it. thank for catching it. I have updated now

Please extract this into its own test method.

Well, that's now questionable...

This looks truly inspired.

I find that surprising. I'd expect `Django <em>pony</em>` or a `SuspiciousOperation`.

Good idea: #20409

Thanks! Took a glimpse and saw it was nontrivial, will try to review this week.

Sure thing -- no rush!