SQLite schema introspection should use PRAGMA instead of parsing to retrieve foreign keys Description The SQLite backend's DatabaseIntrospection.getrelations uses a complex combination of regexes to extract foreign key constraints from DDL while PRAGMA foreignkeylist works just fine as getforeignkey_constraints.


Internationalisation didn't support language locale containing both script and region. Description The i18npatterns didn't work with locale contains both script and region, like en-latn-us. Given LANGUAGECODE = 'en-us' LANGUAGES = [ ('en-us', "English"), ('en-latn-us', "Latin English"), ('en-Latn-US', "BCP 47 case format"), ] from import i18npatterns from import HttpResponse def bangiah(request): return HttpResponse('U!') urlpatterns += i18npatterns( path('', bangiah), ) The response of http://localhost:8000/en-us/ is 200 U!. The response of http://localhost:8000/en-lat-us/ is 404 not found. The response of http://localhost:8000/en-Latn-US/ is 404 not found. Steps to Reproduce Start a new project with django-admin startproject tshi and cd tshi/ Append to as follows LANGUAGES = [ ('en-us', "English"), ('en-latn-us', "Latin English"), ('en-Latn-US', "BCP 47 case format"), ] MIDDLEWARE += [ '', ] Edit by appending follows from import i18npatterns from import HttpResponse def bangiah(request): return HttpResponse('U!') urlpatterns += i18npatterns( path('', bangiah), ) python migrate python runserver The results The response of http://localhost:8000/en-us/ is 200 U!. The response of http://localhost:8000/en-lat-us/ is 404 not found. The response of http://localhost:8000/en-Latn-US/ is 404 not found. Expect to happen instead The response of http://localhost:8000/en-latn-us/ and http://localhost:8000/en-Latn-US/ should be 200 U!. The en-Latn-US tag follows format defined in RFC 5646. It's documented that the language part is always in lowercase, following Accept-Language. Accept-Language is following Content-Language Header, which is following RFC 5646.


The RFC 5646 defined langtag as follow: langtag = language ["-" script] ["-" region] ("-" variant) ("-" extension) ["-" privateuse] language = 23ALPHA ; shortest ISO 639 code ["-" extlang] ; sometimes followed by ; extended language subtags / 4ALPHA ; or reserved for future use / 58ALPHA ; or registered language subtag extlang = 3ALPHA ; selected ISO 639 codes *2("-" 3ALPHA) ; permanently reserved script = 4ALPHA ; ISO code region = 2ALPHA ; ISO 3166-1 code / 3DIGIT ; UN M.49 code I have confirmed that this issue can be reproduced as described on a fresh Django project Python version: 3.7.5 Django version: 3.2.7


startproject should honor umask Description Ticket fixed the problem of not being made executable by copying all the permission bits (not just the executable flags). This means that the user's umask doesn't work, e.g.: $ umask 077 $ touch foo $ ls -l foo -rw------- 1 talex talex 0 2007-05-12 13:27 foo $ PYTHONPATH=trunk ./trunk/django/bin/django- startproject mysite $ ls -l -rw-r--r-- 1 talex talex 2804 2007-05-12 13:28 I discovered this whilst trying to make a Zero Install package for Django. Everything in the Zero Install cache is read-only, so startproject fails with: File "", line 799, in startproject fp = open(mainsettingsfile, 'w') IOError: [Errno 13] Permission denied: '' Thanks,


Make the elementid argument of jsonscript optional Description I recently had a use-case where I wanted to use json_script but I didn't need any id for it (I was including the <scriptinside a <templateso I didn't need an id to refer to it). I can't see any reason (security or otherwise) for the id to be required and making it optional doesn't seem to break any tests.


KeyError with migration autodetector and FK field with hardcoded reference Description Hi, I encountered this issue on an old Django project (probably 10 years old) with tons of models and probably a lot of questionable design decisions. The symptom is that running our test suite in verbose mode doesn't work: $ python test -v 2 Creating test database for alias 'default' ('testproject')... Operations to perform: Synchronize unmigrated apps: [... about 40 apps] Apply all migrations: (none) Synchronizing apps without migrations: Creating tables... Creating table authpermission Creating table authgroup Creating table authuser Creating table djangocontenttype Creating table djangosession Creating table djangoadminlog [... 100 or so more tables] Running deferred SQL... Running migrations: No migrations to apply.


Traceback (most recent call last): File "", line 17, in <moduleexecutefromcommandline() File "", line 401, in executefromcommandline utility.execute() File "", line 395, in execute self.fetchcommand(subcommand).runfromargv() File "", line 23, in runfromargv super().runfromargv(argv) File "", line 330, in runfromargv self.execute(args, cmdoptions) File "", line 371, in execute output = (args, options) File "", line 53, in handle failures = testrunner.runtests(testlabels) File "", line 697, in runtests oldconfig = self.setupdatabases(aliases=databases) File "", line 618, in setupdatabases self.parallel, kwargs File "", line 174, in setupdatabases serialize=connection.settingsdict['TEST'].get('SERIALIZE', True), File "", line 77, in createtestdb runsyncdb=True, File "", line 168, in callcommand return command.execute(args, defaults) File "", line 371, in execute output = (args, options) File "", line 85, in wrapped res = handlefunc(args, kwargs) File "", line 227, in handle changes = autodetector.changes() File "", line 43, in changes changes = self.detectchanges(convertapps, graph) File "", line 160, in detectchanges self.generaterenamedmodels() File "", line 476, in generaterenamedmodels modelfieldsdef = self.onlyrelationagnosticfields() File "", line 99, in onlyrelationagnosticfields del deconstruction[2]['to'] KeyError: 'to' I finally did some digging and found that the culprit is a custom ForeignKey field that hardcodes its to argument (and thus also removes it from its deconstructed kwargs). It seems that the autodetector doesn't like that.


Here's a self-contained reproduction test to replicate the issue: from import models from import MigrationAutodetector from import ModelState, ProjectState from import TestCase class CustomFKField(models.ForeignKey): def init(self, args, kwargs): kwargs['to'] = 'testapp.HardcodedModel' super().init(*args, kwargs) def deconstruct(self): name, path, args, kwargs = super().deconstruct() del kwargs["to"] return name, path, args, kwargs class ReproTestCase(TestCase): def testreprodution(self): before = ProjectState() before.addmodel(ModelState('testapp', 'HardcodedModel', [])) after = ProjectState() after.addmodel(ModelState('testapp', 'HardcodedModel', [])) after.addmodel(ModelState('testapp', 'TestModel', [('custom', CustomFKField(ondelete=models.CASCADE))])) changes = MigrationAutodetector(before, after).detectchanges() self.assertEqual(len(changes['testapp']), 1) While I'll happily admit that my custom field's design might be questionable, I don't think it's incorrect and I think the autodetector is at fault here. Changing del deconstruction[2]['to'] to deconstruction[2].pop('to', None) on the line indicated by the traceback makes my test suite run again, in all its glorious verbosity. Seems like an innocent enough fix to me.


Window(orderby) should allow usage of descending string syntax to be used Description (last modified by Simon Charette) The QuerySet.orderby and some aggregates ordering kwarg allows for the leading dash syntax to be used but Window.orderby doesn't as it solely wraps the provided orderby in ExpressionList(expressions=orderby). This makes for an inconsistent API so I suggest we reuse the logic in in As a related note it seems most of the logic of OrderableAggMixin could be simplified by using ExpressionList. It's a shame that we used ordering and not orderby as a kwarg for OrderableAggMixin as it's now inconsistent. Also not sure how much of a public API the OrderBy expression is but I wish it was initially named Sort (or Ordering?) so that we could define class OrderBy(ExpressionList): template = 'ORDER BY %(expressions)s' def init(self, expressions, extra): expressions = [ (Sort(F(expr[1:]), descending=True) if isinstance(expr, str) and expr[0] == '-' else expr) for expr in expressions ] super().init(expressions, *extra) And then simply use this abstraction in Window and Postgres orderable aggregates. Assigning to myself as I plan to have a look at this in next few days.


django-admin start[project|app] doesn't send proper user agent header when HTTP(S) url is sent. Description (last modified by rsp2k) By default, when fetching a remote template, django-admin start[app|project] uses the default urllib User Agent which causes some sites to block requests (namely gitlab, since they use cloudflare See ). This patch sets the 'User Agent' header to Django/DJANGO_VERSION.


LEVELTAGS not updated when using Description When reading messages inside tests, new message tags created using is not updated. That causes the property results to be an empty string and not know the new tags.


Query.changealiases raises an AssertionError Description Python Version: 3.9.2 Django Version: 2.2.24, 3.2.9 (reproduced using two different versions) Code to Reproduce from import models class Foo(models.Model): qux = models.ForeignKey("app.Qux", ondelete=models.CASCADE, relatedname="foos") class Bar(models.Model): foo = models.ForeignKey("app.Foo", ondelete=models.CASCADE, relatedname="bars") anotherfoo = models.ForeignKey("app.Foo", ondelete=models.CASCADE, relatedname="otherbars") baz = models.ForeignKey("app.Baz", ondelete=models.CASCADE, relatedname="bars") class Baz(models.Model): pass class Qux(models.Model): bazes = models.ManyToManyField("app.Baz", relatedname="quxes") from import Q from import Foo, Qux qux = () qs1 = () qs2 = ( Q(()) | Q(()) ) qs2 | qs1 qs1 | qs2 Description I have encountered this bug during working on a project, recreated the code to reproduce as simple as I can. I have also examined the reason behind this bug, as far as I understand the reason is that during an or operation of two QuerySets, in Query.combine method of the variable combined, if rhs's Query currently have sequential aliases (e.g. T4 and T5) and related tablenames also exist in lhs.tablemap, calling Query.tablealias in will result in creation of aliases T5 for T4 and T6 for T5, thus changemap's keys intersect with changemap's values, so the AssertionError above is raised. Expectation Could you please fix this bug? Maybe aliasmap of rhs can be provided to and Query.tablealias, and suffix (number) of the new alias might be incremented until it is not in rhs.aliasmap, to prevent intersection between changemap's keys and values. Assertion in the first line of QuerySet.changealiases is not documented via a comment. As far as I understand, it is there because if keys and values intersects it means that an alias might be changed twice (e.g. first T4 -T5, and then T5 -T6) according to their order in the changemap. IMHO there can be a comment about what it assures, or an explanation can be to the AssertionError (like the assertions in the Query.combine method).


It seems like QuerySet's OR operation is not commutative (they can create different queries, even though the results are the same), IMHO this can be explicitly declared on the documentation.


Saving parent object after setting on child leads to unexpected data loss in bulkupdate(). Description Consider following example: class Child(models.Model): pass class Parent(models.Model): child = models.ForeignKey(Child, ondelete=models.CASCADE, null=True) parent = (child=None) = Child() () Parent.objects.bulkupdate([parent], fields=["child"]) Expected behavior: parent model instance was updated with the ID to the child in the database. Actual behavior: Parent model is still referencing Null. There should probably be some check for ForeignKeys in the bulkupdate logic, and if one is updated, the ID of the child model should be re-copied to the child_id field that is actually being written to the database.


Deprecate PickleSerializer and move it out of core Description Pickle serializer has long been known to be dangerous. This is mitigated by requiring MAC on pickle in cookies, but nevertheless, RCEs continue to happen:  To further discourage it's use, we should consider deprecating PickleSerializer and moving it into a third party package.


Detecting uniqueness doesn't work for models with functional unique constraints. Description When creating a new object from Django Administration site, I'm getting an error "Tag with this already exists.". (Tag is my model name) I'm using on this model the new functional unique constraints introducted in Django 4.0 (). I suppose this is related to the contraints put on the model. However, creating a Tag object with code - e.g. (...) - works fine. It only fails in Django admin. I'm not sure if it's a bug or misunderstanding/misconfiguration on my side? This is my model: class Tag(models.Model): """Basic tag model.""" # When using getorcreate, we need to pass tag name as a default value, like this: # Tag.objects.getorcreate(defaults={'name': tag}, nameiexact=tag, ) user = models.ForeignKey(settings.AUTHUSERMODEL, ondelete=models.CASCADE, verbosename=('user')) name = models.CharField(maxlength=50, dbindex=False, verbosename=('name'), blank=False, # It's the default behavior, but let's be explicit validators=[RegexValidator(regex=r'[A-z0-9À-ž\s]+', message='This field accepts only letters, digits and space.')]) slug = models.SlugField(maxlength=50, verbosename=('slug')) class Meta: ordering = ['name'] # Make sure tag name: # - are unique per user # - are case insensitive (prevent adding "test" if "Test" already exists) # - aren't empty constraints = [models.UniqueConstraint(fields=['user', 'name'], name='%(applabel)s%(class)snameuniqueperuser'), models.UniqueConstraint(Lower('name'), name='%(applabel)s%(class)snamecaseinsensitive'), models.CheckConstraint(check=models.Q(namelengthgt=0), name='%(applabel)s%(class)snamenotempty')] (snip) This is the admin configuration: class TagAdmin(admin.ModelAdmin): listdisplay = ('pk', 'user', 'name', 'slug') listdisplaylinks = ['pk'] fields = ('user', 'name',) listfilter = ('useremail',) search_fields = ('name',)


Use simplified paths for deconstruct of expressions Description Previously F() deconstructed to: (). But since it can also be imported from , PR changed it to deconstruct to (). This simplifies generated migration code where it will be referenced only as from import models / models.F(). As Mariusz pointed out on the PR, the same technique can be applied to other expressions, further simplifying generated migrations.


DatabaseCache backend doesn't quote all fields in queries Description Snowflake requires all fields to be quoted, otherwise they're treated as uppercase. The attached patch works with stable/3.2.x, but I'll have to review it once django-snowflake development is caught up to Django's main branch.


path()/repath() should raise a TypeError when kwargs is not a dict. Description Apparently, however many years into using Django, I'm still capable of making a "newbie" mistake and getting confused. So perhaps other actual new users encounter similar, especially given the lack of typing specifiers. I defined a URL like so: urlpatterns = [ path("path/to/thing", (), "myview"), ] which ... well, you either spot the issue immediately or you don't, and end up with the following. If you try and resolve() the path (eg: by making a request in your browser), you'll get something like: In [3]: resolve("/path/to/thing") in resolve(path, urlconf) 22 if urlconf is None: 23 urlconf = geturlconf() ---24 return getresolver(urlconf).resolve(path) 25 26 in resolve(self, path) 586 for pattern in self.urlpatterns: 587 try: --588 submatch = pattern.resolve(newpath) 589 except Resolver404 as e: 590 self.extendtried(tried, pattern, e.args[0].get('tried')) in resolve(self, path) 388 newpath, args, kwargs = match 389 # Pass any extrakwargs as kwargs. --390 (self.defaultargs) 391 return ResolverMatch(self.callback, args, kwargs, , route=str(self.pattern)) 392 ValueError: dictionary update sequence element has length 1; 2 is required The crux of the issue being that I meant to give the URL a name, and it's a super unfortunate history that kwargs comes before the name argument (because nearly everyone gives a URL a name, but passing static kwargs is comparatively infrequent). So what's actually happened is that kwargs = "myview" and eventually self.defaultargs = "myview".


If I update to path("path/to/thing", (), "myview", name="myview"), leaving the type incorrect, I can get the following error via reverse, too: In [4]: reverse("myview") in reverse(viewname, urlconf, args, kwargs, currentapp) 84 resolver = getnsresolver(nspattern, resolver, tuple(())) 85 ---86 return resolver.reversewithprefix(view, prefix, args, kwargs) 87 88 in reversewithprefix(self, lookupview, prefix, args, **kwargs) 669 if set(kwargs).symmetricdifference(params).difference(defaults): 670 continue --671 if any((k, v) != v for k, v in ()): 672 continue 673 candidatesubs = kwargs AttributeError: 'str' object has no attribute 'items' Both of these suggest that either there should be a type-guard in path to assert it's dict-ish (if not None), or a system check on URLPattern to raise a friendly message. Well, they actually continue to suggest to me that everything after the view argument should be keyword-only, or that kwargs should come later, but I suspect those to be a harder sell ;) This is specifically around the kwargs, but it doesn't look like there's any guarding on the name either, and I feel like a name of {'test': 'test'} (i.e. accidentally swapped both positionals) is likely to bite & cause an issue somewhere.


Deprecate cachedproperty's name argument Description Django 2.2 is the last version to support Python 3.5 where cachedproperty's name argument is required. Following the release of Django 4.0, most apps will drop support for Django 2.2 (and hence Python 3.5), so Django 4.0 can deprecate the name argument without giving warnings that aren't actionable as long as apps want to keep support for Python 3.5.


URLField throws ValueError instead of ValidationError on clean Description forms.URLField( ).clean('////] results in: ValueError: Invalid IPv6 URL Traceback (most recent call last): File "", line 22, in TestOneInput File "", line 350, in testformsURLField File "", line 151, in clean File "", line 136, in runvalidators File "", line 130, in call File "", line 440, in urlsplit


fails to handle broken data Description (last modified by Florian Apolloner) The actual input string was 'P3(3D' === Uncaught Python exception: === ValueError: could not convert string to float: '3(3' Traceback (most recent call last): File "", line 22, in TestOneInput File "", line 294, in testformsDurationField File "", line 149, in clean File "", line 502, in topython File "", line 154, in parse_duration File "", line 154, in <dictcomp


nevercache()/cachecontrol() decorators raise error on duck-typed requests. Description The cache decorators cachecontrol, nevercache and sensitivepostparameters no longer work with Django REST framework because they strictly check for an HttpRequest instance.


ExpressionWrapper for ~Q(pkin=[]) crashes. Description (last modified by Stefan Brand) Problem Description I'm reducing some Q objects (similar to what is described in ticket:. Everything is fine for the case where the result is ExpressionWrapper(Q(pkin=[])). However, when I reduce to ExpressionWrapper(~Q(pkin=[])) the query breaks. Symptoms Working for ExpressionWrapper(Q(pkin=[])) print(queryset.annotate(foo=ExpressionWrapper(Q(pkin=[]), outputfield=BooleanField())).values("foo").query) SELECT 0 AS "foo" FROM "table" Not working for ExpressionWrapper(~Q(pkin=[])) print(queryset.annotate(foo=ExpressionWrapper(~Q(pkin=[]), outputfield=BooleanField())).values("foo").query) SELECT AS "foo" FROM "table"


Delete nonexistent migrations from djangomigrations table Description Django adds a djangomigrations table to the database which list all of the migrations that have been applied (and when). With the introduction of squashmigrations, it is possible for this table to contain a lot of old migrations that no longer exist. This can be problematic if naming duplication occurs: Example: I have an app with: myapp/migrations/ I squash and delete replaced migrations: myapp/migrations/ I create a new migration and use poor naming: myapp/migrations/ My new migration never runs because the djangomigrations table thinks it has already been applied. I propose truncation of the djangomigrations table so that it includes only migrations that actually exist in the django project. This could be done automatically (when executor runs, or inside the migrate mcommand). Or have its own mcommand that requires it be run manually. I prefer the automatic approach though. Pros: Cleans up old data that just bloats the database. Protects users from the trap mentioned above where a new migration is created with the same name as one that was applied in the past. Cons: A loss of historical information. Note: Need to be careful with implementation to avoid a possible new trap if a user squashes migrations and then proceeds to delete the replaced migrations before running the squashed migrations on their database -django will think squashed migrations haven't been applied and will attempt to reapply them. This can be remedied simply by not removing migrations mentioned in replaces lists of other migrations from djangomigrations (ie. we'd consider replaced migrations as still existing, even if their actual files have already been removed).


Add ability to know where an object was deleted from to pre/post delete signals Description Since deleting objects deletes related objects, it would be good to know the origin of the delete. It lets the signal listeners know why an object is being delete. If it's deleted from a (), you know which instance initiated the delete and if it's from a () you know which queryset initiated the delete, with the models in that queryset. Using this you know if the instance is being deleted directly or because it's related to another instance.


MigrationRecorder does not obey dbrouter allowmigrate rules Description Hi, We have a multi-db setup. We have one connection that is for the django project, and several connections that talk to other dbs for information (ie models with managed = False). Django should only create tables in the first connection, never in any of the other connections. We have a simple router that does the following: class Router(object): def allowmigrate(self, db, model): if db == 'default': return True return False Current Behaviour We run our functional tests and the migrate command is called against each connection when the test databases are created (see , setupdatabases, line 300-ish, which calls , createtestdb, line 377-ish) When this migrate runs, it tries to apply our migrations, which tries to record that a migration has been applied (see , applymigration, which has several calls to self.recorder.recordapplied). The first thing that recordapplied does is a call to self.ensureschema() (see , recordapplied, lien 66-ish). ensureschema checks to see if the Migration model is in the tables in the connection. If it does not find the table then it tries to create the table. I believe that this is incorrect behaviour when a dbrouter has been provided. If using the router above, my expectation would be that the table is not created on any connection other than the 'default' connection. Looking at the other methods on the MigrationRecorder, I would expect that there will be similar issues with appliedmigrations and record_unapplied.


Optimize multiple AlterFooTogether operations into one Description Hi, In we split the AlterFooTogether (AlterUniqueTogether and AlterIndexTogether) operations into two types of operations. First, a migration will have operations to remove constraints, and then other operations adds the new constraints. This allows field alterations to work as expected during in between operations. In some cases, this introduced two operations that can actually easily be reduced to one. See for instance the test case:  Example: operations = [ migrations.AlterUniqueTogether( name='mymodel', uniquetogether=set(), ), migrations.AlterIndexTogether( name='mymodel', indextogether=set(), ), migrations.AlterUniqueTogether( name='mymodel', uniquetogether={("col",)}, ), migrations.AlterIndexTogether( name='mymodel', indextogether={("col",)}, ), ] should be optimized to operations = [ migrations.AlterUniqueTogether( name='mymodel', uniquetogether={("col",)}, ), migrations.AlterIndexTogether( name='mymodel', indextogether={("col",)}, ), ] So that we don't do two operations on each constraint, but only one.


Add a management command to optimize a migration Description (last modified by Raphael Gaschignard) Because the migration optimizer still has a bit of trouble with reducing operations, you often want to edit squashed migrations and re-run it through the optimizer. The attached patch contains an implementation of a management command, optimizemigration, that will help speed up this process. optimizemigration appname migrationname reads a single migration, pass it through the optimizer, and then rewrite the result to disk. Unlike squashmigrations, this ignores things like the squashed-ness of the migration.


Micro-optimisation for Value.resolveoutputfield (by modifying ) Description Currently, when you do something like annotate(x=Value('test')) that will eventually probably call down into Value.resolveoutputfield() and run the following code: if isinstance(, str): return fields.CharField() which is innocuous enough. However, CharField currently expects that self.maxlength is always a non null value of sensible data, and AFAIK this is caught for users at system-check time as a requirement for use. So what currently happens is that the CharField internally gets granted a MaxLengthValidator which cannot work and must be demonstrably extraneous (i.e. validators aren't used the outputfield, at least for Value) x = Value('test') y = x.resolveoutputfield() y.validators [ at 0x105e3d940] y.clean('1', modelinstance=None) in compare(self, a, b): TypeError: '' not supported between instances of 'int' and 'NoneType' Further compounding this is that MaxLengthValidator is decorated by (both directly and indirectly via BaseValidator ...?). So, baseline (as of ): In [1]: from import Value In [2]: x = Value('test') In [3]: %timeit x.resolveoutputfield() 1 µs ± 39.6 ns per loop (mean ± std. dev. of 7 runs, loops each) (Note: a previous run was faster at 7.6µs, so normal CPU workfload flux is in effect). We can see how much of the time is because of (see my comment here on a PR about deconstructible being a source to potentially optimise away) by just commenting it out from both validator classes: In [1]: from import Value In [2]: x = Value('test') In [3]: %timeit x.resolveoutputfield() 96 µs ± 130 ns per loop (mean ± std. dev. of 7 runs, loops each) But ignoring the class instantiation altogether is faster, easier and more correct at this juncture: In [1]: from import Value In [2]: x = Value('test') In [3]: %timeit x.resolveoutputfield() 86 µs ± 45.4 ns per loop (mean ± std. dev.


of 7 runs, loops each) So roughly a 2µs improvement. How do we get to that? Change the to: if self.maxlength is not None: (validators.MaxLengthValidator(self.max_length)) which incidentally and happily is the same process taken by for precedent. I have a branch locally with this change, and all existing tests currently pass. I'll push it to CI once I get a ticket number out of this submission, and see if it causes any issues elsewhere, and we can decide if it can be accepted from there.


Adding nullable OneToOneField crashes on SQLite. Description This new sqlite3 error has cropped up between building django-oauth-toolkit between Django 4.0 and main branch for migrations.AddField of a OneToOneField (see ): self = object at 0x10b8038b0query = 'ALTER TABLE "oauth2provideraccesstoken" ADD COLUMN "sourcerefreshtokenid" bigint NULL UNIQUE REFERENCES "oauth2providerrefreshtoken" ("id") DEFERRABLE INITIALLY DEFERRED' params = [] def execute(self, query, params=None): if params is None: return Database.Cursor.execute(self, query) query = self.convertquery(query) E Cannot add a UNIQUE column Here's the relevant migration snippet: migrations.AddField( modelname='AccessToken', name='sourcerefreshtoken', field=models.OneToOneField(blank=True, null=True, , to=oauth2settings.REFRESHTOKENMODEL, relatedname="refreshedaccess_token"), ), I see there have been a lot of sqlite3 changes in since the 4.0 release....


Deferred fields incorrect when following prefetches back to the "parent" object Description Given the following models: class User(models.Model): email = models.EmailField() kind = models.CharField( maxlength=10, choices=[("ADMIN", "Admin"), ("REGULAR", "Regular")] ) class Profile(models.Model): fullname = models.CharField(maxlength=255) user = models.OneToOneField(User, ondelete=models.CASCADE) I'd expect the following test case to pass: def testonlyrelatedqueryset(self): user = ( email="", kind="ADMIN", ) (user=user, fullname="Test Tester") queryset = ("email").prefetchrelated( Prefetch( "profile", queryset=Profile.objects.prefetchrelated( Prefetch("user", ("kind")) ), ) ) with self.assertNumQueries(3): user = () with self.assertNumQueries(0): self.assertEqual(, "ADMIN") The second assertNumQueries actually fails with: AssertionError: 1 != 0 : 1 queries executed, 0 expected Captured queries were: "testsuser"."id", "testsuser"."kind" FROM "testsuser" WHERE "testsuser"."id" = 1 This is exactly the query I'd expect to see if kind on the inner User queryset had been deferred, which it hasn't. The three queries executed when iterating the main queryset (ie when executing user = ()) look correct: "testsuser"."id", "testsuser"."email" FROM "testsuser" ORDER BY "testsuser"."id" ASC LIMIT 1 "testsprofile"."id", "testsprofile"."fullname", "testsprofile"."userid" FROM "testsprofile" WHERE "testsprofile"."userid" IN (1) "testsuser"."id", "testsuser"."kind" FROM "testsuser" WHERE "testsuser"."id" IN (1) Printing () returns {'kind'}. It looks to me like Django is correctly evaluating the set of deferred fields when executing the "inner" User queryset, but somehow the instances are inheriting the set of fields they "think" have been deferred from the outer User queryset, so when the attribute is accessed it causes a database query to be executed. It appears that this also happens if the relationship between Profile and User is a ForeignKey rather than a OneToOneField (in that case, a query is executed when accessing ()[0]).


I'm happy to attempt to tackle this if someone can (a) confirm it's actually a bug and (b) point me in the right direction! Thanks :)


Technical 404 debug page reported incorrect view name for CBVs. Description Technical 404 debug page reported incorrect view name for CBVs, e.g. Raised by: <class ''instead of Raised by: Regression in . Thanks Keryn Knight for the report.


ResolverMatch repr is incorrect for Class Based Views Description The following test applies cleanly to 3.2.9, and AFAIK would apply roughly correctly all the way back to when CBVs were introduced (I can't easily test without going back to a super old Python and finding the test cases, which have moved around): """ add to class: """ def testclassbasedrepr(self): self.assertEqual( repr(resolve('/redirect/')), "ResolverMatch(, " "args=(), kwargs={}, urlname=None, appnames=[], " "namespaces=[], route=redirect/)", ) The funcpath as AFAIK always been a representation to the fully qualified dotted callable where possible, that is for a CBV it's the CBV module + the class name. As of 4.0, the funcpath has become , I believe because of removing the use of updatewrapper and intentionally not setting the name and qualname in favour using the viewclass attribute, as per the comment viewclass should be used to robustly determine the name of the view (see Pull Request) Unfortunately I think that means the detection of class based views in ResolverMatch no longer works correctly, and this can probably only be resolved by making ResolverMatch CBV aware again by embedding detection of viewclass therein. Noted it as a question in this PR for ticket , but hoisting it here properly to be considered separately. The fix appears to ostensibly the same as for (see PR) class ResolverMatch: def init(...): # ... if hasattr(func, 'viewclass'): func = func.viewclass if not hasattr(func, 'name'): # ... I have a branch which I shall push shortly to confirm it works generally.


Model () should be immutable. Description changes value when a field is assigned to a model class. This code crashes with an AssertionError: from import models f = models.CharField(max_length=200) d = {f: 1} class Book(models.Model): title = f assert f in d The bug was introduced in . It's unlikely to have been encountered because there are few use cases to put a field in a dict before it's assigned to a model class. But I found a reason to do so whilst implementing and the behaviour had me stumped for a little. IMO we can revert the hash change from . Objects with the same hash are still checked for equality, which was fixed in that ticket. But it's bad if an object's hash changes, since it breaks its use in dicts.


simplifyregex() doesn't handle non-capturing groups Description (last modified by Mariusz Felisiak) While using Django REST Framework's Schema generator, I found out they're using simplifyregex(); however, current version has a few shortcomings, namely non-capturing groups are broken. simplifyregex() doesn't handle non-capturing groups Description (last modified by Mariusz Felisiak) While using Django REST Framework's Schema generator, I found out they're using simplifyregex(); however, current version has a few shortcomings, namely non-capturing groups are broken.


Unify reverse foreign key and m2m unsaved model querying Description Currently when querying unsaved reverse relations the behavior differs. Using model: class Foo(models.Model): fk = models.ForeignKey('self', relatedname='fkrev') m2m = models.ManyToManyField('self') and test case: print(Foo()()) print(Foo()()) We get [] from the first filter, but an error ValueError: "<Foo: Foo object" needs to have a value for field "fromfoo" before this many-to-many relationship can be used. from the second filter. So, m2m fields can't be filtered if the object isn't saved, but reverse fk fields can be filtered. There is a (slightly stale) patch for which makes fk fields and m2m fields work consistently. The changes in behavior are: Nullable many-to-many and foreign key relations will return an empty queryset when the relation field is null. For many-to-many this was previously an error (no change for foreign keys). Trying to add objects to a foreign key relation when the relation field is null is now an error (no change for m2m). Trying to use a relation of any kind when the object isn't saved is now an error (no change for m2m). I think we can squeeze these changes in as bug-fixes. These are slight backwards compatibility changes, but to me it seems that almost always the changes will be visible only in code that isn't working as intended. If these are seen as something likely breaking working code, then I don't see any way to make the APIs consistent. The patch is available from:


() generates invalid SQL. Description (last modified by M1ha Shvn) Since this commit (...) method returns incorrect SQL removing first and last symbols instead of absent breakets. Adding Subquery().query.subquery = True attribute fixes the problem. From my point of view, it should be set in Subquery constructor. from import connection from import App q = Subquery(()) print(str(q.query)) print(q.assql(q.query.getcompiler('default'), connection)) q.query.subquery = True print(q.assql(q.query.getcompiler('default'), connection)) ('(SELECT "appsapp"."id", "appsapp"."name" FROM "apps_app")', ())


File upload crash when a file extension contains null characters. Description A 2.5M file uploaded with a raw null byte anyplace after the . in its filename means that Django attempts to create a tempfile with that same "extension," which errors out with ValueError: embedded null byte. It's almost certainly a violation of RFC to have a filename with a raw null byte in it, but it shouldn't result in a 500 when parsing the form. Here's code to generate a bad request: #!/usr/bin/env python3 import io import requests contents = io.StringIO("." (1024 1024 * 3)) files = {"docfile": (b"!", contents, "text/plain")} req = requests.Request("POST", "http://localhost:8000/", files=files, data={}) prepared = req.prepare() body = assert isinstance(body, bytes) = body.replace(b"!", b"\x00") requests.Session().send(prepared) produces an error with the view: from django import forms from import HttpResponseRedirect from django.shortcuts import render from import csrfexempt class UploadFileForm(forms.Form): docfile = forms.FileField() def index(request): if == 'POST': form = UploadFileForm(request.POST, request.FILES) if form.is_valid(): print(repr(request.FILES['docfile'])) return HttpResponseRedirect('/') else: print("Not valid!") return HttpResponseRedirect('/') else: form = UploadFileForm() return render(request, '', {'form': form}) I'm not sure what the goal is of preserving the "extension" of the uploaded file in the tempfile that is made; if that's important enough a behaviour to keep, some amount of escaping on the parsed-out extension may be necessary.


Support prefetchrelated() with Queryset.iterator() Description (last modified by Asif Saifuddin Auvi) I was surprised when I found out that prefetchrelated calls are ignored when using Queryset.iterator. I noticed in the docs here  that it is because the "these two optimizations do not make sense together." That may have been true in the past, but it is definitely not the case now. The iterator allows chunking (by default 2000) and it would be very helpful to prefetch related for each chunk.


Migration autodetector crashes on models with field named order, but not using orderwithrespectto. Description (last modified by Fabian Büchler) The commit  introduced a new function ModelState.getfield in This converts the field name order to the one defined in options['orderwithrespectto'] automatically, which fails if the model has a field order but isn't using Meta.orderwithrespectto. That is the case for models generated by django-simple-history () for models that are originally using Meta.orderwithrespectto: the resulting historical records model has only order but is not using the Meta option.


This shows when running migrate or makemigrations: $ makemigrations --dry-run Waiting for port 'mysql:3306' timeout 1s (attempt 1/60) Port 'mysql:3306' is open Traceback (most recent call last): File "", line 42, in <modulemain() File "", line 36, in main executefromcommandline() File "/usr/local/lib/python3.8/site-", line 425, in executefromcommandline utility.execute() File "/usr/local/lib/python3.8/site-", line 419, in execute self.fetchcommand(subcommand).runfromargv() File "/usr/local/lib/python3.8/site-", line 373, in runfromargv self.execute(args, cmdoptions) File "/usr/local/lib/python3.8/site-", line 417, in execute output = (args, options) File "/usr/local/lib/python3.8/site-", line 90, in wrapped res = handlefunc(args, *kwargs) File "/usr/local/lib/python3.8/site-", line 172, in handle changes = autodetector.changes( File "/usr/local/lib/python3.8/site-", line 43, in changes changes = self.detectchanges(convertapps, graph) File "/usr/local/lib/python3.8/site-", line 189, in detectchanges self.generatealteredfields() File "/usr/local/lib/python3.8/site-", line 928, in generatealteredfields oldfield = [applabel, oldmodelname].getfield(oldfieldname) File "/usr/local/lib/python3.8/site-", line 689, in getfield self.options['orderwithrespectto'] KeyError: 'orderwithrespectto' I believe this could be solved using a bit more defensive code, like: def getfield(self, fieldname): if fieldname == 'order' and 'orderwithrespectto' in self.options: fieldname = self.options['orderwithrespectto'] return [fieldname]


Messages framework incorrectly serializes/deserializes extratags when it's an empty string Description When a message is serialised and then deserialised with any of the built in storage backends, then extratags=="" is converted to extratags==None. This is because MessageEncoder checks for the truthyness of extratags rather than checking it is not None. To replicate this bug from import settings settings.configure() # Just to allow the following import from import Message from import MessageEncoder, MessageDecoder originalmessage = Message(10, "Here is a message", extratags="") encodedmessage = MessageEncoder().encode(originalmessage) decodedmessage = MessageDecoder().decode(encodedmessage) originalmessage.extratags == "" True decodedmessage.extratags is None True Effect of the bug in application behaviour This error occurred in the wild with a template tag similar to the following: {% if x not in message.extratags %} When the message was displayed as part of a redirect, it had been serialised and deserialized which meant that extratags was None instead of the empty string. This caused an error. It's important to note that this bug affects all of the standard API (, etc. all have a default value of extra_tags equal to "").


Session ID should be cleansed from error reporting Description the session ID should be cleansed when reporting errors, just like other credentials. A patch is available at  See also and  A quick github search yielded multiple occasions where session IDs ended up in public bug reports:     I am sure you could find many more. This could potentially be exploited by automatically searching for such requests and hijacking the associated accounts.


bulkupdate() does not work with plain F('...') expressions. Description Repro: assign plain F(...) to some model instance field save with bulkupdate Example: Code highlighting: from import SelfRef o = ().first() o.c8 = F('name') # model has char fields 'c8' and 'name' SelfRef.objects.bulkupdate([o], ['c8']) 1 o.refreshfromdb() o.c8 'F(name)' from import connection connection.queries[-2] {'sql': 'UPDATE "exampleappselfref" SET "c8" = CASE WHEN ("exampleappselfref"."id" = ) THEN \'F(name)\' ELSE NULL END WHERE "exampleappselfref"."id" IN ()', 'time': '0.001'} The created SQL contains the string repr of F(), instead of resolving to the column name. Looking at the source code, the culprit seems to be a too narrow type check in  It works, if the type check gets replaced by one of these: Code highlighting: if not hasattr(attr, 'resolve_expression'): ... if not isinstance(attr, (Expression, F)): ...


Introduce empty slots protocol for SafeString & SafeData Description (last modified by Keryn Knight) This is a case-by-case proposal ultimately referencing Because SafeString is used a lot and is otherwise supposed to be treatable as a untainted str we should be able to (AFAIK) update it + it's inheritance chain to use slots = () whilst still allowing custom subclasses of either to add additional attributes. By defining slots as empty on SafeString (and SafeData) we'd avoid creation of a dict on the instance, which mirrors the str() behaviour. According to pympler, currently in Python 3.10 using the following back of the napkins strings: In [4]: s = "test" # this might be interned, as a short string? In [5]: s2 = "test" 100 In [6]: s3 = SafeString("test") In [7]: s4 = SafeString("test" 100) we get: In [8]: asizeof(s) # str Out[8]: 56 In [9]: asizeof(s2) # str Out[9]: 456 In [10]: asizeof(s3) # SafeString Out[10]: 208 In [11]: asizeof(s4) # SafeString Out[11]: 608 But if we swap out the implementation to be slots'd, it looks more like: In [8]: asizeof(s) # str Out[8]: 56 In [9]: asizeof(s2) # str Out[9]: 456 In [10]: asizeof(s3) # SafeString Out[10]: 104 In [11]: asizeof(s4) # SafeString Out[11]: 504 So we're "saving" 104 bytes per SafeString created, by the look of it. I presume it to be some fun implementation detail of something somewhere that it is allegedly accounting for more than 64 bytes, which is the asizeof({}) A quick and dirty check over the test suite suggests that for me locally, running tests in 512.912s accounted for 949.0 MB of SafeStrings, checked by just incrementing a global integer of bytes (using and --parallel=1) and piping that to filesizeformat, so y'know, room for error.


After the patch, the same tests accounted for 779.4 MB of SafeString, "saving" 170 MB overall. The only functionality this would preclude -- as far as I know -- is no longer being able to bind arbitrary values to an instance like so: s = SafeString('test') s.test = 1 which would raise AttributeError if slots were , just like trying to assign attributes to str() directly does. I don't believe this will have any marked performance change, as neither SafeString nor SafeData actually have any extra attributes, only methods. I have a branch which implements this, and tests pass for me locally.


aggregate() with 'default' after annotate() crashes. Description I saw this on a PostgreSQL project and reproduced it with SQLite. Django 4.0.1. Annotate (anything) then aggregate works fine: $ shell Python 3.10.2 (main, Jan 21 2022, 19:45:54) [Clang 13.0.0 (clang-1300.0.29.30)] Type 'copyright', 'credits' or 'license' for more information IPython 7.30.1 -- An enhanced Interactive Python. Type '?' for help. In [1]: from import In [2]: from import In [3]: from import * In [4]: () Out[4]: 95 In [5]: Book.objects.annotate(idx=F("id")).aggregate(Sum("id")) Out[5]: {'idsum': 4560} But add the aggregate classes’ default argument (new in 4.0), and it breaks: In [6]: Book.objects.annotate(idx=F("id")).aggregate(Sum("id", default=0)) OperationalError Traceback (most recent call last) ... OperationalError: near "FROM": syntax error The generated SQL: In [7]: %debug 414 return Database.Cursor.execute(self, query) 415 query = self.convertquery(query) --416 return Database.Cursor.execute(self, query, params) 417 418 def executemany(self, query, paramlist): ipdbquery 'SELECT FROM (SELECT "corebook"."id" AS "idx", COALESCE(SUM("corebook"."id"), ?) AS "idsum" FROM "core_book") subquery' ipdbparams (0,) ipdbThe “long form” using Coalesce works: In [8]: Book.objects.annotate(idx=F("id")).aggregate(x=Coalesce(Sum("id"), 0)) Out[8]: {'x': 4560}


Migration autodetector crashes when renaming a model and field. Description Migration autodetector crashes when renaming a model and field in a single step: $ python makemigrations Did you rename the testone.MyModel model to MyModel2? [y/N] y Traceback (most recent call last): File "", line 22, in <modulemain() File "", line 18, in main executefromcommandline() File "", line 419, in executefromcommandline utility.execute() File "", line 413, in execute self.fetchcommand(subcommand).runfromargv() File "", line 354, in runfromargv self.execute(args, cmdoptions) File "", line 398, in execute output = (args, options) File "", line 89, in wrapped res = handlefunc(args, *kwargs) File "", line 172, in handle changes = autodetector.changes( File "", line 43, in changes changes = self.detectchanges(convertapps, graph) File "", line 182, in detectchanges self.generaterenamedfields() File "", line 823, in generaterenamedfields newmodelstate = [applabel, oldmodelname] KeyError: ('testone', 'mymodel') Reported by HoskeOwl. Regression in .


filter on exists-subquery with empty queryset removes whole WHERE block Description (last modified by Tobias Bengfort) qs = (~models.Exists(()), name='test') qs <QuerySet []print() EmptyResultSet With django-debug-toolbar I can still see the query, but there WHERE block is missing completely. This seems to be very similar to .


Dev Server fails to restart after adding BASEDIR to TEMPLATES[0]['DIRS'] in settings Description Repro steps: $ pip install -U django $ django-admin startproject <nameOpen , copy the BASEDIR variable from line 16 and paste it into the empty DIRS list on line 57 $ runserver Back in your IDE, save a file and watch the dev server NOT restart. Back in , remove BASE_DIR from the templates DIRS list. Manually CTRL-C your dev server (as it won't restart on its own when you save), restart the dev server. Now return to your file, re-save it, and notice the development server once again detects changes and restarts. This bug prevents the dev server from restarting no matter where you make changes - it is not just scoped to edits to


SimpleLazyObject doesn't implement radd Description Technically, there's a whole bunch of magic methods it doesn't implement, compared to a complete proxy implementation, like that of wrapt.ObjectProxy, but radd being missing is the one that's biting me at the moment. As far as I can tell, the implementation can't just be radd = newmethodproxy() because that doesn't exist, which is rubbish. radd = newmethodproxy(operator.attrgetter("radd")) also won't work because types may not have that attr, and attrgetter doesn't supress the exception (correctly) The minimal implementation I've found that works for me is: def radd(self, other): if self.wrapped is empty: () return other + self._wrapped


Deprecate passing unsaved objects to related filters. Description Consider this filter: (relatedobj=bar) Where 'bar' is an unsaved object instance. In Django 1.11, this would always return an empty QuerySet (since no Foo object is related to unsaved 'bar'). In Django 2.0 through 2.2, this is equivalent to doing (which can return a non-empty QuerySet): (relatedobj=None) I found a somewhat related issue that touches on this subject: My questions: What is the intended behaviour? In the aforementioned issue Simon Charette suggests that unsaved objects should be prevented from being used in related filters. I agree with that. Is this documented anywhere? I couldn't find anything. At the very least this should be documented somewhere.


"Synchronous middleware ... adapted" is not logged for synchronous middleware Description This is a bug in log messages that contradicts the guide in a warning in the docs; the feature works. The guide at , emphasis mine: Warning You will only get the benefits of a fully-asynchronous request stack if you have no synchronous middleware loaded into your site. If there is a piece of synchronous middleware, then Django must use a thread per request to safely emulate a synchronous environment for it. Middleware can be built to support both sync and async contexts. Some of Django’s middleware is built like this, but not all. To see what middleware Django has to adapt, you can turn on debug logging for the django.request logger and look for log messages about “Synchronous middleware … adapted”. The test for "Synchronous middleware ... adapted" is instead testing an async middleware over a sync method:  'middlewareexceptions.middleware.asyncpaymentmiddleware', ]) def testasyncmiddleware(self): with self.assertLogs('django.request', 'DEBUG') as cm: response = ('/middlewareexceptions/view/') self.assertEqual(response.statuscode, 402) self.assertEqual( cm.records[0].getMessage(), "Synchronous middleware " "middlewareexceptions.middleware.asyncpaymentmiddleware " "adapted.", ) About the existing implementation: BaseHandler.loadmiddleware passes the middleware name as name to BaseHandler.adaptmethodmode:  adaptedhandler = self.adaptmethodmode( middlewareisasync, handler, handlerisasync, debug=settings.DEBUG, name='middleware %s' % middlewarepath, ) BaseHandler.adaptmethodmode adapts the method and treats name as the method name rather than the middleware name; when the middleware name is used, it implies a method has been adapted for the middleware, not that the middleware was adapted: if debug and not name: name = name or 'method %s()' % method.qualname if isasync: if not methodisasync: if debug: ('Synchronous %s adapted.', name) return synctoasync(method, threadsensitive=True) elif methodisasync: if debug: ('Asynchronous %s adapted.


', name) return asynctosync(method) Proposed fix: Handle middleware name and method name separately within BaseHandler.adaptmethodmode: def adaptmethodmode( self, isasync, method, methodisasync=None, debug=False, name=None, ): """ Adapt a method to be in the correct "mode": If isasync is False: Synchronous methods are left alone Asynchronous methods are wrapped with asynctosync If isasync is True: Synchronous methods are wrapped with synctoasync() Asynchronous methods are left alone """ methodname = None if methodisasync is None: methodisasync = asyncio.iscoroutinefunction(method) if debug and not name: name = name or 'method %s()' % method.qualname methodname = 'method %s()' % method.qualname if isasync: if not methodisasync: if debug: ('Synchronous %s adapted.', name) if name: ('Asynchronous %s adapted.', name) else: ('Synchronous %s adapted.', methodname) return synctoasync(method, threadsensitive=True) elif methodisasync: if debug: ('Asynchronous %s adapted.', name) if name: ('Synchronous %s adapted.', name) else: ('Asynchronous %s adapted.', methodname) return asynctosync(method)


QuerySet.bulkupdate() should perform atomic writes against routed dbforwrite. Description Discovered in but the summary is that bulkupdate doesn't set self.forwrite = True prior to accessing so the latter is actually dbforread and that includes the block creation.


Allow parallel test runner to work with Windows/macOS process start method. Description (last modified by Brandon Navra) Python 3.8 on MacOS has changed the default start method for the multiprocessing module from fork to spawn:  When running tests with the --parallel flag, this causes the worker processes to fail with Apps aren't loaded yet. as they no longer have a copy of the parent memory state. It can also cause the workers to fail to find the cloned dbs ( {{ FATAL: database "xxx1" does not exist}} ) as the db test prefix is missing. I have attached a patch which changes (the worker initialiser for ParallelTestSuite) to run () and set the db name to one with the test prefix.


LazyObject defines attribute that don't exist on wrapped object Description LazyObject defines magic methods (getitem, iter) which may be missing from the wrapped object. This leads to the following errors: somevariable = if hasattr(somevariable, "getitem"): foo = somevariable["foo"] # raises TypeError: 'User' object has no attribute 'getitem' if hasattr(somevariable, "iter"): for item in some_variable: # raises TypeError: 'User' object is not iterable


ManyToManyField to lowercased swappable setting causes generating infinite migrations. Description If I create a custom user model that extends AbstractUser and then try to add a ManyToManyField that references this custom User model, django keeps making the same AlterField migration over and over again unnecessarily. I've attached a Git repository that I used to reproduce this issue with very simple code. You can see the two erroneous migrations after the initial migration. If I use the built in user, there is no issue. It seems to appear once I extend AbstractUser. Git repository:  Here is the accounts app code. from import settings from import AbstractUser from import models class User(AbstractUser): pass class Test(models.Model): members = models.ManyToManyField(settings.AUTHUSERMODEL) Here is one of the erroneous migrations. Notice it depends on 0002altertestmembers which is an erroneous migrations file that looks just like this one. from import settings from import migrations, models class Migration(migrations.Migration): dependencies = [ ('accounts', '0002altertestmembers'), ] operations = [ migrations.AlterField( modelname='test', name='members', field=models.ManyToManyField(to=settings.AUTHUSER_MODEL), ), ]


Fallback to a more generic language variant for Select2 translations. Description When using for example the setting LANGUAGECODE="de-ch", the autocompletefield widget will be in English as select2 does not provide the specific regional translation for "de-ch". However the existing translation language "de" would be a much better approximation. I suggest using the language without region from the LANGUAGE_CODE in case an exact match can not be found. This is not a duplicate of .


forces evaluation of lazy objects Description Consider the following example: from import marksafe from import activate, ugettextlazy as s = marksafe(("username")) tpl = Template('{{ s }}') activate('fr') print((Context({'s': s}))) I would expect this to output nom d'utilisateur (which is the french translation of username) but what happens instead is that it outputs username. The reason for this is that marksafe will force the evaluation of the lazy string provided by ugettextlazy when it's called. Unfortunately, the solution to this it trickier than simply wrapping marksafe with , because marksafe can operate both on bytes and text (and allowlazy needs to know the type of object return by the wrapped function). I wrote some tests and a proposed solution on my branch:


ModelAdmin with defined radiofields override emptylabel Description ModelAdmin drops my "emptylabel" and set "defaultemptylabel". For example: class MyModelAdmin(ModelAdmin): radiofields = 'myfield', def formfieldforforeignkey(self, dbfield, args, kwargs): if == 'myfield': kwargs['emptylabel'] = "I WANT TO SET MY OWN EMPTY LABEL" return super().formfieldforforeignkey(dbfield, args, kwargs) You get never the "I WANT TO SET MY OWN EMPTY LABEL" How to fix it: In django\contrib\admin\, row 234: kwargs['emptylabel'] = ('None') if else None Should be changed on: kwargs['emptylabel'] = (('emptylabel') or ('None')) if else None


Unapplying ContentType migration 0002 does not populate legacy name field on non-default database Description Given a database with the alias other that has all contenttypes migrations applied, roll the state back to initial: $ migrate --database=other contenttypes 0001 All ContentType rows in the other database will be null instead of their intended value.


setcookie and setsignedcookie should accept timedelta object for maxage argument Description This already works for getsignedcookie: request.getsignedcookie("mykey", maxage=timedelta(days=3)) This is due to the underlying behaviour of TimestampSigner, which was fixed to do this in . But for setcookie and setsignedcookie it accepts only a number: response = HttpResponse() response.setcookie("mykey", maxage=timedelta(days=3)) TypeError: int() argument must be a string, a bytes-like object or a number, not 'datetime.timedelta'


AppAdmin class for customizing app listing in admin index Description See  for details. This supplements the app directive. As discussed with brosner and jkocherhans in #django-dev: <brosnerit looks reasonable, but haven't spent much time thinking about it <jkocherhansmrts: I think this is clearly backwards incompatible with the current nfa api and has to go in pre 1.0 if it goes in at all <jkocherhansI'm a big -1 on the order attribute and -0 on models (maybe just a different syntax), but the other stuff seems reasonable <mrtsjkocherhans: what's wrong with ordering? <jkocherhansit just feels like the wrong place to specify it <jkocherhansit's a global issue, and an issue any particular app should handle <mrtsmy use case: I have a lot of functionality exposed to somewhat dumb users <mrtsand they have trouble finding the right bits in the admin interface ordering is only used in context of admin index I would like to put the important apps to top and collapse the rest <jkocherhansexactly. what should 3rd party apps put there? therein lies my objection. <mrtswell, I'd say decouple admin from models (as nfa already does) and don't specify any admin options at all -- users are free to customize things with AppAdmin <jkocherhansI guess not if using a AppAdmin class is optional. I was originally thinking it would replace model registration with an admin site. <mrtsjkocherhans: yeah, that's what I kinda meant... it looks more coherent this way jkocherhans: and it may solve some of the issues register() currently has <jkocherhansmrts: I'm gonna have to let it sit for awhile. I'm trying to think of what else an AdminApp class would do besides being a coathanger for a few attributes, nothing is coming to mind. <mrtsjkocherhans: but jezdez has a point -- it would also provide easy bridging for app instances Example syntax follows.


class BarModelAdmin(admin.ModelAdmin): description = 'A bar is a bar is a bar' ... class FooAppAdmin(admin.AppAdmin): app = settings.INSTALLED_APPS[0] name = "Foo" # overrides app().name description = "An application that does foo" style = {'classes' : ('collapse',)} order = 1 models = ( # model order in this list determines their display order in app block (BarModel, BarModelAdmin), (BazModel, None), # use default ModelAdmin, don't show description ) (FooAppAdmin) # no need for the tedious for model in [A, B, C, D]: (model)


Builtin csrffailure() view uses wrong charset Description When Django detects wrong CSRF token, it shows an error using view That file ends with return HttpResponseForbidden(t.render(c), contenttype="text/html;") When the template (CSRFFAILURETEMPLATENAME) is written using UTF-8, it is rendered incorrectly. I suggest changing that line to return HttpResponseForbidden(t.render(c), contenttype="text/html;"+ f" charset={settings.DEFAULTCHARSET};") or perhaps leaving out the content_type entirely. Currently I'm using a workaround, by adding <meta http-equiv="Content-type" content="text/html; charset=utf-8" /to the template's HEAD, but it seems to me that the suggested fix is a better solution.


Allow syncing user attributes on every authentication with RemoteUserBackend. Description If using a custom RemoteUserBackend, it is possible to want to synchronize any changes from the remote system back into the Django user records whenever authentication happens. Currently, if any user attributes change in the remote system there is no easy way to reflect these changes back into the users in the Django system. The goal of this feature is to introduce a new method in the class called synchronizeuser with a method signature equal to that of configureuser (which it complements) that will be called in the authenticate method of said class right after fetching the user from the database (if any), regardless of whether the user was unknown and created or otherwise. Implementors can then override this method and implement data synchronization between the remote user and the Django user that will be applied on every user authentication attempt.


Add support for multiple values for the x-forwarded-proto header Description When Django is deployed behind more than one proxy, the proxy behavior is sometimes to list the protocol as a comma-separated list. However, currently, Django expects only one value for the x-forwarded-proto header, instead of parsing it as a list of values and setting the protocol accordingly. x-forwarded-proto is a non-standard header, so there isn't a specification for its use, but different reverse-proxy vendors do use it in different ways, and some append the protocol as a comma-separated value from left-to-right (left being the furthermost proxy and rightmost being the closest). Similar issues have been raised and implemented in other projects, for example: Tornado: Issue:  Implementation:  Ruby: Issue:  Implemenation:  Reactor-Netty:  Implementation:  Most implementation use the leftmost-value or rightmost value. I would expect that provided that you are certain that the initial proxy can be trusted, that the left-most value makes the most sense, since it represent the original value at the entry-point for the HTTP request which is often where TLS is being terminated. Common example of this behavior is when using mulitple AWS proxies such as API Gateway proxying to an elastic load balancer.


Fix handling empty string for If-Modified-Since header Description Empty string used to be ignored for If-Modified-Since header, but now raises exception since . Fix handling empty string for If-Modified-Since header Description Empty string used to be ignored for If-Modified-Since header, but now raises exception since .


Optimize CreateModel + AlterModelManagers to CreateModel Description During migration optimization, CreateModel + AlterModelOptions is reduced to just CreateModel, with the model options. Similarly, CreateModel + AlterModelManagers can become just CreateModel.


haskey, haskeys, and hasanykeys JSONField() lookups don't handle numeric keys on SQLite, MySQL, and Oracle. Description (last modified by TheTerrasque) Problem When using models.JSONField() haskey lookup with numerical keys on SQLite database it fails to find the keys. Versions: Django: 4.0.3 Python: 3.9.6 (tags/v3.9.6:, Jun 28 2021, 15:26:21) [MSC v.1929 64 bit (AMD64)] on win32 sqlite3.version: '2.6.0' sqlite3.sqliteversion: '3.35.5' Example: Database DATABASES = { 'default': { 'ENGINE': '', 'NAME': 'db.sqlite3', } } Model class JsonFieldHasKeyTest(models.Model): data = models.JSONField() Test from import TestCase from .models import JsonFieldHasKeyTest class JsonFieldHasKeyTestCase(TestCase): def setUp(self) -None: test = JsonFieldHasKeyTest(data={'foo': 'bar'}) test2 = JsonFieldHasKeyTest(data={'1111': 'bar'}) () () def testjsonfieldhaskey(self): c1 = (datahaskey='foo').count() c2 = (datahaskey='1111').count() self.assertEqual(c1, 1, "Should have found 1 entry with key 'foo'") self.assertEqual(c2, 1, "Should have found 1 entry with key '1111'") Result FAIL: testjsonfieldhaskey () Traceback (most recent call last): File "H:\Files\Projects\Electaco\Webservice\elecserve\markers\", line 16, in testjsonfieldhaskey self.assertEqual(c2, 1, "Should have found 1 entry with key '1111'") AssertionError: 0 != 1 : Should have found 1 entry with key '1111' Additional info This has been tested on SQLite and Postgresql backend, it works on postgresql but fails on sqlite.


makemessages command skips the nl-nl-x-informal folder Description There is a mismatch on the DjangoTranslation folder and the makemessages command for the value nl-nl-x-informal DjangoTranslation uses the tolocale method to determine the language folder to read the tolocale translates nl-nl-x-informal correctly to the nlNL-x-informal folder. However makemessages skips the nlNL-x-informal folder and displays the following message invalid locale nlNL-x-informal, did you mean nlNLxinformal?  The check for - in the locale should only be for the first section a.k.a. nl_NL


loaddata fails on non-default database when natural keys uses foreign keys. Description (last modified by François Granade) I've got a one-to-many relationship between two models Book and Author, that define a natural keys in both models. I'm loading some data from a fixture. It works in the default database, but when I use it a second database, then I get an exception. I'm relatively new to natural keys and to serializers, but I wouldn't expect things to work differently in the default DB and others ?


I've committed a test project here:  (The problem doesn't appear if the data is already present in the default DB) The error: % cat | loaddata --database other --format json - Traceback (most recent call last): File "/Users/francois/Library/Caches/pypoetry/virtualenvs/exportbug-PGt-cwXF-py3.9/lib/python3.9/site-", line 187, in get relobj = (instance) File "/Users/francois/Library/Caches/pypoetry/virtualenvs/exportbug-PGt-cwXF-py3.9/lib/python3.9/site-", line 15, in getcachedvalue return [cachename] KeyError: 'author' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/francois/Library/Caches/pypoetry/virtualenvs/exportbug-PGt-cwXF-py3.9/lib/python3.9/site-", line 70, in Deserializer yield from PythonDeserializer(objects, options) File "/Users/francois/Library/Caches/pypoetry/virtualenvs/exportbug-PGt-cwXF-py3.9/lib/python3.9/site-", line 174, in Deserializer obj = base.buildinstance(Model, data, using) File "/Users/francois/Library/Caches/pypoetry/virtualenvs/exportbug-PGt-cwXF-py3.9/lib/python3.9/site-", line 332, in buildinstance naturalkey = Model(data).naturalkey() File "/Users/francois/lmad/src/django-", line 33, in naturalkey return (,) + () File "/Users/francois/Library/Caches/pypoetry/virtualenvs/exportbug-PGt-cwXF-py3.9/lib/python3.9/site-", line 205, in get relobj = self.getobject(instance) File "/Users/francois/Library/Caches/pypoetry/virtualenvs/exportbug-PGt-cwXF-py3.9/lib/python3.9/site-", line 168, in getobject return ((instance)) File "/Users/francois/Library/Caches/pypoetry/virtualenvs/exportbug-PGt-cwXF-py3.9/lib/python3.9/site-", line 496, in get raise ( Author matching query does not exist.


the model: from import models class AuthorManager(models.Manager): def getbynaturalkey(self, name): return (name=name) class Author(models.Model): id = models.AutoField(primarykey=True) name = models.CharField(maxlength=255, unique=True) objects = AuthorManager() def naturalkey(self): return (,) def str(self): return f"{} {}" class BookManager(models.Manager): def getbynaturalkey(self, title, author): # OR title, author ?? return (title=title, authorname=author) class Book(models.Model): id = models.AutoField(primarykey=True) title = models.CharField(maxlength=255) author = models.ForeignKey(Author, models.DONOTHING, relatedname="books") objects = BookManager() def naturalkey(self): return (,) + () naturalkey.dependencies = ["testbug.Author"] class Meta: uniquetogether = [["title", "author"]] def str_(self): return f"{}: '{}' by {}" the data (generated with from import serializers; from import Book, Author; print(serializers.serialize("json", list(()) + list(()), indent=2, usenaturalforeignkeys=True, usenaturalprimary_keys=True)) in the shell): [ { "model": "", "fields": { "name": "JR Tolkien" } }, { "model": "", "fields": { "title": "The Ring", "author": [ "JR Tolkien" ] } } ]


viewonsite redirect does not work for custom admin site. Description All reverse calls in have a currentapp keyword argument, so they resolve with the current (possibly custom) admin site – except the reverse call in getviewonsiteurl. This lead to custom admin sites using the default admin to redirect via `viewon_site. This is clearly a bug. When the default admin is handled with extra protections outside of Django (e.g. additional basic auth), users of the custom admin cannot use the "view on site" link because it routes through the default admin. Patch is easy and already here:  I'll try to provide a regression test as well.


Using multiple FilteredRelation with different filters but for same relation is ignored. Description (last modified by lind-marcus) I have a relation that ALWAYS have at least 1 entry with isall=True and then I have an optional entry that could have isall=False but instead have zone set. I'm trying to use FilteredRelation together with Case(When()) to ensure that it use the zone level one (if exist) and fall back on "All" if zone do not exist. from import FilteredRelation ( relationzone=FilteredRelation( "myrelationnested", condition=Q(myrelationnestedzone=F("zone")) ), relationall=FilteredRelation( "myrelationnested", condition=Q(myrelationnestedisall=True) ), pricezone=F("relationzoneprice") ).annotate( pricefinal=Case( When( pricezoneisnull=True, then=F("relationallprice"), ), default=F("pricezone") ) ) I noticed that when using multiple FilteredRelation with the same relation (myrelationnested) it actually just generates a single SQL JOIN (if inspecting the raw SQL) and ignores the other. So in this case if I do print(str()) I would only see a join for relationzone. Not for relation_all. Is this intended behavior or should I be able to do the thing above?


createsuperuser does not validate usernames that use a UniqueConstraint. Description With a custom User model that uses a UniqueConstraint instead of unique=True, the createsuperuser command does not validate usernames at all. class CustomUser(AbstractBaseUser): customusername = models.CharField(maxlength=255) USERNAMEFIELD = 'customusername' class Meta: constraints = [ models.UniqueConstraint(fields=['customusername'], name='uniqueusernames'), ] Running createsuperuser to create a user with a username that already exists then results in an IntegrityError: IntegrityError: duplicate key value violates unique constraint "uniqueusernames" DETAIL: Key (customusername)=(foo) already exists.


AlterField operation should be noop when adding/changing choices on SQLite. Description while writing a test case for i found that for sqlite, even a seemingly db-transparent change like adding choices still generates sql (new table + insert + drop + rename) even though this shouldn't be needed. on e.g. postgres the same migration generates no sql


Wrong behavior on queryset update when multiple inheritance Description Queryset update has a wrong behavior when queryset class inherits multiple classes. The update happens not on child class but on other parents class instances. Here an easy example to show the problem: class Base(models.Model): baseid = models.AutoField(primarykey=True) fieldbase = models.IntegerField() class OtherBase(models.Model): otherbaseid = models.AutoField(primarykey=True) fieldotherbase = models.IntegerField() class Child(Base, OtherBase): pass Then in django shell: In [1]: (fieldotherbase=100) <QuerySet [{'otherbaseid': 1, 'fieldotherbase': 100}]In [2]: (fieldotherbase=101) <QuerySet [{'otherbaseid': 2, 'fieldotherbase': 101}]In [3]: (fieldbase=0, fieldotherbase=0) <Child: Child object (1)In [4]: (fieldbase=1, fieldotherbase=1) <Child: Child object (2)In [5]: (fieldotherbase=55) SELECT "applianceschild"."baseptrid" FROM "applianceschild" Execution time: 0.000647s [Database: default] UPDATE "appliancesotherbase" SET "fieldotherbase" = 55 WHERE "appliancesotherbase"."otherbaseid" IN (1, 2) Execution time: 0.001414s [Database: default] Out[5]: 2 In [6]: ('fieldotherbase') <QuerySet [{'fieldotherbase': 0}, {'fieldotherbase': 1}]In [7]: (otherbaseidin=[1,2]).values('fieldotherbase') <QuerySet [{'fieldotherbase': 55}, {'fieldotherbase': 55}]As seen on the above code, updating Child fields from second parent has no effect. Worse is that OtherBase fields where modifed because query is using primiary keys from Base class.


RegisterLookupMixin.unregisterlookup() should clear the lookup cache. Description (last modified by Himanshu Balasamanta) In current source code, in the unregisterlookup method, , the cache is not cleared, which should be done, as it is done in registerlookup,  Corresponding to this change, minor changes need to be brought in the test. The PR generated is


Django 3.2.4+ autoreload breaks on empty string in TEMPLATES DIRS. Description Django versions 3.2.3 changes the way template dirs are handled, they are now normalized using pathlib.Path. People having an invalid value in TEMPLATESDIRS? will notice that autoreload stops working. "DIRS": ("TEMPLATESDIRS", "").split(",") # wrong, should be filter(None, ("TEMPLATESDIRS", "").split(",")) or anything else that produces this: "DIRS": [''] # wrong will break autoreload. This happens because was previously comparing the empty string to a directory, and would never match. Now the normalization transforms the empty string into the root of the project. The result is that templatechanged() will now always return True, preventing the autoreload when the app code changes Change that produced the regression Commits in main and stable/3.2.x:   Previous reports


clear select fields for of distinct, non-sliced querysets Description The fix () for bug introduced an inefficiency in .exists() on distinct querysets in that now all distinct querysets are selecting all their fields. I propose the fields should be left on the queryset only if (q.distinct and (q.highmark is not None or q.lowmark is not None)). As I started writing the test for this, I realized I would have to inspect the query itself, which seems... like a bad idea for a test. Would this be reasonable to submit the (one-line-change) patch without a test, or does someone have a suggestion for a way to make the test that's less brittle?


Enable cached template loader in development. Description (last modified by Carlton Gibson) Following the changes to allow auto-reloading cached templates in , there was discussion on PR to enable the cached template loader by default even in development. (Folks not wanting that would specify loaders.)


Unnecessary redirect in LogoutView when ?next=... contains "unsafe" URL Description Reproduction instructions Set LOGOUTREDIRECTURL Wire () at /logout/ in the URLconf Add this form to any template: <form action="/logout/?next=http://evil/" method="POST"<input type="submit" value="Logout"{% csrftoken %}</formLog in, then use the form to log out Expected result You are logged out; the next parameter is ignored; you are redirected to LOGOUTREDIRECTURL Actual result There is an intermediary, useless redirect; see the logs of the development server: [16/Apr/2022 19:05:38] "POST /logout/?next=http://evil/ HTTP/1.1" 302 0 [16/Apr/2022 19:05:38] "GET /logout/ HTTP/1.1" 302 0 [16/Apr/2022 19:05:38] "GET /en/ HTTP/1.1" 200 I noticed this via code inspection. The implementation of LogoutView.getnextpage seemed a bit weird to me. This stems from  which predates the introduction of LOGOUTREDIRECTURL. From the user's perspective, the behavior is correct. There's just an extra round-trip and needlessly complicated code.


Inconsistent / Unexpected handling of assigning unsaved model to Generic Foreign Key Description addresses the issue of assigned an unsaved model to a ForeignKey or OneToOneField (raises error when save() called), however the same logic / pattern does not apply to GFKs. Given: class ModelA(models.Model): name = models.CharField(maxlength=20) class ModelB(models.Model): gfkctype = models.ForeignKey(ContentType, ondelete=models.PROTECT) gfkid = models.PositiveIntegerField() gfk = GenericForeignKey('gfkctype', 'gfkid') class ModelC(models.Model): fk = models.ForeignKey(ModelA, ondelete=models.CASCADE) Foreign Key Behaviour: In [2]: a = ModelA(name='Model A') In [3]: c = ModelC(fk=a) In [4]: c.fk Out[4]: <ModelA: ModelA object (None)In [5]: c.save() ... ValueError: save() prohibited to prevent data loss due to unsaved related object 'fk'. In [6]: a.save() (0.016) INSERT INTO "testappmodela" ("name") VALUES ('Model A'); args=['Model A'] In [7]: c.fk Out[7]: <ModelA: ModelA object (1)In [8]: c.save() (0.016) INSERT INTO "testappmodelc" ("fkid") VALUES (1); args=[1] GFK behaviour: In [9]: a2 = ModelA(name='Model A2') In [10]: b = ModelB(gfk=a2) In [11]: b.gfk Out[11]: <ModelA: ModelA object (None)In [12]: b.save() (0.000) INSERT INTO "testappmodelb" ("gfkctypeid", "gfkid") VALUES (9, NULL); args=[9, None] IntegrityError: NOT NULL constraint failed: In [14]: () (0.015) INSERT INTO "testappmodela" ("name") VALUES ('Model A2'); args=['Model A2'] In [15]: b.gfk (0.000) SELECT "testappmodela"."id", "testappmodela"."name" FROM "testappmodela" WHERE "testappmodela".


"id" IS NULL LIMIT 21; args=() None In [17]: b.gfkctype Out[17]: <ContentType: testapp | model aTwo observations: No check on b.gfk and b.gfkid value during save() which could lead to silent data loss if b.gfkid is nullable. When a2 is saved, accessing b.gfk now does a redundant DB query to try and find ModelA instance with PK = None, then then returns None value (effectively un-assigning a2 model), while keeping b.gfkctype intact. This is because the new pk of a2 is different to the existing gfkid (pkval) of the GFK field (None) What should be done: Modify () or Model.preparerelatedfieldsforsave() to also perform verification check for GFK fields Modify () to handle case of pkval = None (update fkfield value using PK value of GFK model if present, do not perform redundant DB query on pk=None, return previously assigned (then saved) model instead of None)


Add SQL comment to describe deliberately no-op migration operations Description Currently when a field migration is a no-op, the operation description is output in SQL, but nothing else. This can be confusing as to which operations are no-ops. It could be clearer if we output an extra SQL comment when there are deliberately no statements to execute for a given migration operation. Take for example this output: BEGIN; -- Alter field name on Author ALTER ...; -- Alter field title on Book COMMIT; The field has an operation applied, whilst needs no changes to the database. This isn't exactly clear from the output - is the COMMIT part of the change? It could be clearer as: BEGIN; -- Alter field name on Author ALTER ...; -- Alter field name on Author -- (no-op) COMMIT; (Or perhaps more verbose wording, like "no SQL to execute") I think this can help especially when there are consecutive operations with no-op SQL: BEGIN; -- Alter field name on Author -- (no-op) -- Alter field name on Author -- (no-op) COMMIT; (Inspired by , where the OP suggested dropping such migration operation header comments.)


Errors with dbcollation – no propagation to foreignkeys Description (last modified by typonaut) Using dbcollation with a pk that also has referenced fks in other models causes foreign key constraint errors in MySQL. With the following models: class Account(models.Model): id = ShortUUIDField(primarykey=True, dbcollation='utf8bin', dbindex=True, maxlength=22) … class Address(models.Model): id = ShortUUIDField(primarykey=True, dbcollation='utf8bin', dbindex=True, maxlength=22) account = models.OneToOneField(Account, ondelete=models.CASCADE) … class Profile(models.Model): id = ShortUUIDField(primarykey=True, dbcollation='utf8bin', dbindex=True, maxlength=22) … account = models.ForeignKey('Account', verbosename=('account'), null=True, blank=True, ondelete=models.CASCADE) … etc Where has been changed from models.BigAutoField if makemigrations is run then it produces sqlmigrate output like this: ALTER TABLE MODIFY varchar(22) COLLATE ; ALTER TABLE MODIFY varchar(22) NOT NULL; ALTER TABLE MODIFY varchar(22) NULL; ALTER TABLE ADD CONSTRAINT FOREIGN KEY () REFERENCES (); ALTER TABLE ADD CONSTRAINT FOREIGN KEY () REFERENCES (); With this SQL the ADD CONSTRAINT queries fail. This is because the COLLATE should also be present in the bmanageaddress.accountid and bmanageprofile.accountid modification statements. Like this: ALTER TABLE MODIFY varchar(22) COLLATE ; ALTER TABLE MODIFY varchar(22) NOT NULL COLLATE ; ALTER TABLE MODIFY varchar(22) NULL COLLATE ; ALTER TABLE ADD CONSTRAINT FOREIGN KEY () REFERENCES (); ALTER TABLE ADD CONSTRAINT FOREIGN KEY () REFERENCES (); In the latter case the ADD CONSTRAINT statements run without error. The collation of the pk must match the collation of the fk otherwise an error will occur.


Unnecessary column in a GROUP BY clause with () Description I've got a question about the interaction between exists() and group by. For example something like: ('field').annotate(cnt=Count('id')).filter(cnt_gt=1).exists() The corresp. query with PostgreSQL looks like this: SELECT (1) AS "a" FROM "appmodel" GROUP BY "appmodel"."field", (1) HAVING COUNT("appmodel"."id") 1 LIMIT 1 exists() (see ) clears the SELECT clause and replaces it by (if I understand correctly) a hardcoded value 1 (as "a"), along with a limit of 1, which makes sense to me. But getgroupby() (see ) pushes this hardcoded value to the GROUP BY clause and we end up with the query above. Now, on PostgreSQL, that works, but to me it sounds like it works by luck/lucky robustness... and certainly the same query without the , (1) in the GROUP BY clause yields the same result. The problem is that outside of PostgreSQL that GROUP BY clause can be downright invalid... Note that if I alter exists() to use {'a': 2} instead of {'a': 1}, it does not work anymore (on PostgreSQL), because (2) in the SELECT clause means the hardcoded number 2 while (2) in the GROUP BY clause presumably (??) refers to the 2nd expr of the SELECT clause, but we only got one... My feeling is that getgroupby() should avoid adding extra/raw sql (or at least extra/raw pure values, if we can detect that...) to the group by expressions? NB: the existing/old ticket that seems most related to my question would probably be:


Migrations changes to implicit primary key when primary key is altered on SQLite. Description Migrations drops explicit primary key defined on a model when it's altered.


views.Feed methods cannot be decorated Description If one applies a decorator on a method which is called by getdynamicattr a TypeError like this occurs: Exception Type: TypeError at /blog/feed/ Exception Value: itemlink() takes exactly 2 arguments (1 given) I think this is because getdynamicattr tries to count the function's arguments, but decorators usally get defined with the args, *kwargs syntax, so this trick does not work here. if code.coargcount == 2: # one argument is 'self' return attr(obj) else: return attr() I think the best approach would be to remove one of the two methods. IMHO We should have either attr(item) or attr() not both, as "there should be one, and preferably only one, obvious way to do it".


Implement RenameIndex in a backwards compatible way Description In order to eventually deprecate indextogether we need a way to deal with old projects that have unnamed indexes. This proves to be a non-trivial problem. Andrew and I came up with these things to consider. RenameIndex(model, newname, oldname=None, oldfields=None) where exactly one of oldname and oldfield is given (oldname ^ oldfields) If the oldname is given we use RENAME INDEX if available Otherwise look at the state and drop existing indexes and create new the index with new name On MySQL (or other DBs) that don't support RENAME INDEX, provide SQL query to look up index name from informationschema by field names and pass in to DROP INDEX. If more than one index is found while identifying with field names, migrations must error out with an AmbiguityError If the autodetector finds an old, unnamed index and a new, named one matching field signature, issue a RenameIndex operation For backwards operations with unnamed old indexes, RenameIndex is a noop.


Query Expression in ordering of a related object fails Description (last modified by Tim Graham) Since 2.0, according to the doc (), we can use QueryExpression objects in the Model.Meta.ordering field. Using: from import models class Musician(models.Model): firstname = models.CharField(maxlength=50) lastname = models.CharField(maxlength=50) instrument = models.CharField(maxlength=100, null=True, blank=True) class Meta: ordering = [models.F('instrument').asc(nullslast=True)] class Album(models.Model): artist = models.ForeignKey(Musician, ondelete=models.CASCADE) name = models.CharField(maxlength=100) releasedate = models.DateField() numstars = models.IntegerField() class Meta: ordering = ['artist'] () ... TypeError: 'OrderBy' does not support indexing When reaching , the compiler tries to use the related model, but at line 679, item can be an OrderBy object. Thus the failure.


Create --update flag for makemigrations management command, mimicking South's one. Description When I was developing with South I was able to use the --update flag for the schemamigration management command () to refine the latest migration. This was very convenient for iterative development. Could we have an equivalent of the --update flag for Django=1.7 makemigrations? (I've taken it from )


Allow overriding FormSet's error messages for too few and too many forms. Description Moving the messages to defaulterrormessages enables overriding them via the errormessages argument when instantiating the FormSet. I would prefer being able to override them via inlineformsetfactory, but that is for another day.


Deprecate CryptPasswordHasher. Description CryptPasswordHasher was 15 years ago mainly to support legacy UNIX apps. It's almost undocumented, not recommended, and supported only on UNIX. Moreover crypt module was deprecated in Python 3.11 (see ). We should deprecate it in Django 4.1 and remove in Django 5.0.


Depracate passing False to OrderBy's nullsfirst and nullslast. Description Consider the following: In [11]: [tv.publishedat for tv in TemplateVersion.objects.orderby(F("publishedat").desc(nullsfirst=True))] Out[11]: [None, datetime.datetime(2022, 2, 25, 13, 0, 12, , tzinfo=<UTC), datetime.datetime(2022, 2, 21, 10, 18, 0, , tzinfo=<UTC)] In [12]: [tv.publishedat for tv in TemplateVersion.objects.orderby(F("publishedat").desc(nullsfirst=False))] Out[12]: [None, datetime.datetime(2022, 2, 25, 13, 0, 12, , tzinfo=<UTC), datetime.datetime(2022, 2, 21, 10, 18, 0, , tzinfo=<UTC)] In [13]: [tv.publishedat for tv in TemplateVersion.objects.orderby(F("publishedat").desc(nullslast=True))] Out[13]: [datetime.datetime(2022, 2, 25, 13, 0, 12, , tzinfo=<UTC), datetime.datetime(2022, 2, 21, 10, 18, 0, , tzinfo=<UTC), None] In [14]: [tv.publishedat for tv in TemplateVersion.objects.orderby(F("publishedat").desc(nullslast=False))] Out[14]: [None, datetime.datetime(2022, 2, 25, 13, 0, 12, , tzinfo=<UTC), datetime.datetime(2022, 2, 21, 10, 18, 0, , tzinfo=<UTC)] Observe how nulls_first=False still puts the nulls first. This happens because they both default False and when they are both False it lets the DB decide. This is surprising behaviour, it also makes changing the null positioning based on a variable more awkward than it needs to be. I think it would be better if they defaulted to None, let the DB decide when both are None and when one is not None do the ordering that implies.


APPENDSLASH adds significant latency to all requests not ending in / (even if successful) Description Originally, APPENDSLASH worked by looking for 404 responses and replacing them with redirects, so as not to unnecessarily impact the performance of successful responses. However, commit in 1.9.5/1.10 changed this to check shouldredirectwithslash() on every request, resulting in a moderately expensive extra urlconf lookup for every request not ending with /, whether or not it succeeds as written. This performance impact was not considered in the commit message or the corresponding ticket , so I assume it was an oversight. That ticket asserted “This doesn't really make sense, since the two settings are not interdependent”, which is incorrect—performance was the reason for the interdependence. The overhead was found to be significant enough in Zulip to merit subclassing CommonMiddleware to skip it in certain conditions. Here’s a minimal test project with an exaggerated number of routes so the overhead can be easily observed. $ runserver $ wrk http://127.0.0.1:8000/url9999 Running 10s test @ http://127.0.0.1:8000/url9999 2 threads and 10 connections Thread Stats Avg Stdev Max +/- Stdev Latency 232.40ms 73.85ms 570.86ms 69.16% Req/Sec 21.70 9.47 40.00 63.35% 426 requests in 10.01s, 64.90KB read Requests/sec: 42.56 Transfer/sec: 6.48KB $ sed -i 's/# APPENDSLASH = False/APPENDSLASH = False/' $ wrk http://127.0.0.1:8000/url9999 Running 10s test @ http://127.0.0.1:8000/url9999 2 threads and 10 connections Thread Stats Avg Stdev Max +/- Stdev Latency 139.80ms 52.07ms 352.19ms 69.09% Req/Sec 36.46 12.23 60.00 58.12% 714 requests in 10.01s, 108.79KB read Requests/sec: 71.32 Transfer/sec: 10.87KB


RenameIndex() crashes when unnamed index is moving backward and forward. Description RenameIndex() should restore the old auto-generated name when an unnamed index for uniquetogether is moving backward. Now re-applying RenameIndex() crashes. For example: diff --git index .. a b class OperationTests(OperationTestBase): with connection.schemaeditor() as editor, self.assertNumQueries(0): operation.databasebackwards(applabel, editor, newstate, projectstate) self.assertIndexNameExists(tablename, "newponytestidx") 2991 # Re-apply renaming. 2992 with connection.schemaeditor() as editor: 2993 operation.databaseforwards(applabel, editor, projectstate, newstate) 2994 self.assertIndexNameExists(tablename, "newponytestidx") # Deconstruction. definition = operation.deconstruct() self.assertEqual(definition[0], "RenameIndex") crashes on PostgreSQL: relation "newponytest_idx" already exists


Templates crash when calling methods for built-in types. Description (last modified by Daniel) Found during a 2.2 -3.2 upgrade: Given a template: {{ foo }} where foo is non-existant, it returns nothing, empty. (That's good) {{ }} also empty (Also good) {% include '' with bar=foo %} and then in having: {{ }} results in File "/Users/daniel/src/django-bug-test/.v/lib/python3.8/site-", line 861, in resolvelookup signature = inspect.signature(current) File "", line 3093, in signature return Signature.fromcallable(obj, followwrapped=followwrapped) File "", line 2842, in fromcallable return signaturefromcallable(obj, sigcls=cls, File "", line 2296, in signaturefromcallable return signaturefrombuiltin(sigcls, obj, File "", line 2107, in signaturefrombuiltin raise ValueError("no signature found for builtin {!r}".format(func)) Exception Type: ValueError at / Exception Value: no signature found for builtin <built-in method count of str object at 0x1100ff2f0On django 2.2, this would not crash, but resulted in empty (as I expected). this seems to fix it for me: diff --git index .. --- +++ -913,15 +913,19 def resolvelookup(self, context): try: # method call (assuming no args required) current = current() except TypeError: signature = inspect.signature(current) try: () except TypeError: # arguments were required current = ( ) # invalid method call signature = inspect.signature(current) except ValueError: # python builtins might not have signature current = else: raise try: () except TypeError: # arguments were required current = ( ) # invalid method call else: raise except Exception as e: templatename = getattr(context, "templatename", None) or "unknown" (


Deprecate Model.Meta.indextogether in favour of Model.Meta.indexes Description Anything that indextogether does can be done by indexes and the Index class.


inspect.signature() returns incorrect signature on manager methods. Description (last modified by Shiva Kumar) inspect.signature returns incorrect signature information when used on queryset methods import inspect from import models class Person(models.Model): name = models.CharField(maxlength=100) print(inspect.signature(Person.objects.bulkcreate)) ipython and jupyter seem to internally use inspect.signature to show documentation when using the <obj? command and they too show incorrect signature information: The issue is due to the code at  Although we are ensuring the decorated method has the right name and docstring on lines 87 and 88, complete metadata is not copied. The fix is to use instead of manually assigning name and docstring. wraps will take care of all the metadata and inspect.signature will return the expected output. If the bug is acknowledged please assign the ticket to me, I would like to raise a PR for this.


Cannot drop uniquetogether constraint on a single field with its own unique=True constraint Description I have an erroneous uniquetogether constraint on a model's primary key (uniquetogether = (('id',),)) that cannot be dropped by a migration. Apparently the migration tries to find all unique constraints on the column and expects there to be only one, but I've got two — the primary key and the uniquetogether constraint: Indexes: "foobarpkey" PRIMARY KEY, btree (id) "foobarid1c3b3088c74c3b17uniq" UNIQUE CONSTRAINT, btree (id) Database is PostgreSQL, if that makes any difference.


Avoid unnecessary clear of cached reference Description (last modified by Barry Johnson) Consider this case of ORM models "Parent" and "Child", where Child has a foreign key reference to Parent (and the database can return generated IDs following insert operations): parent = Parent(name='parentobject') child = Child(parent=parent) () () print() The print statement will cause an unnecessary lazy read of the parent object. In the application where this behavior was first observed, the application was creating thousands of parent and child objects using bulkcreate(). The subsequent lazy reads occurred when creating log entries to record the action, and thousands of unwanted SELECT queries. Closed ticket solved a problem with potential data loss in this situation by essentially executing child.parentid = while preparing the child object to be saved. However, when the child's ForeignKeyDeferredAttrbute "parentid" changes value from None to the parent's ID, the child's internal cache containing the reference to "parent" is cleared. The subsequent reference to then must do a lazy read and reload parent from the database. A workaround to avoid this lazy read is to explicitly update both the "parentid" and "parent" cache entry by adding this non-intuitive statement: = after executing () But it appears that a simple change could avoid clearing the cache in this narrow case. Within Model.preparerelatedfieldsforsave(), replace setattr(self, field.attname, ) with setattr(self, , obj) This suggested code has -not- been tested. This change would set the associated "parent_id" attribute while ensuring that the currently referenced object remains referenced.


Models migration with change field foreign to many and deleting unique together. Description (last modified by Simon Charette) I have models like class Authors(models.Model): projectdataset = models.ForeignKey( ProjectDataSet, ondelete=models.PROTECT ) state = models.IntegerField() startdate = models.DateField() class Meta: uniquetogether = (('projectdataset', 'state', 'startdate'),) and class DataSet(models.Model): name = models.TextField(maxlength=50) class Project(models.Model): datasets = models.ManyToManyField( DataSet, through='ProjectDataSet', ) name = models.TextField(maxlength=50) class ProjectDataSet(models.Model): """ Cross table of data set and project """ dataset = models.ForeignKey(DataSet, ondelete=models.PROTECT) project = models.ForeignKey(Project, ondelete=models.PROTECT) class Meta: uniquetogether = (('dataset', 'project'),) when i want to change field projectdataset in Authors model from foreign key field to many to many field I must delete a uniquetogether, cause it can't be on many to many field. Then my model should be like: class Authors(models.Model): projectdataset = models.ManyToManyField( ProjectDataSet, ) state = models.IntegerField() startdate = models.DateField() But when I want to do a migrations. python3 makemigrations python3 migrate I have error: ValueError: Found wrong number (0) of constraints for appauthors(projectdataset, state, startdate) The database is on production, so I can't delete previous initial migrations, and this error isn't depending on database, cause I delete it and error is still the same. My solve is to first delete unique_together, then do a makemigrations and then migrate. After that change the field from foreign key to many to many field, then do a makemigrations and then migrate. But in this way I have 2 migrations instead of one. I attachment with this project, download it and then do makemigrations and then migrate to see this error.


should allow lazy parameter Description Commit [] (fixing ) triggered a regression when the date template filter (possibly others are affected too) receives a lazy string, like in somedate|date:('Y-m-d'). This fails with: TypeError: getattr(): attribute name must be string in