However, the Combining Dot Above is not a valid \w: [(character, (character), (character)) for character in 'İ'.lower()] [ ('i', 'LATIN SMALL LETTER I', <re.Match object; span=(0, 1), match='i'), # EUREKA!! ('̇', 'COMBINING DOT ABOVE', None) ] So that's why the ValidationError is raised. Proposed Solution The culprit seems to be the order in which lower() is called in slugify. The assumption that the lowercase version of a string is still a valid slugunicodere, does not seem to hold true. Hence, instead of doing this in slugify() (r'[^\w\s-]', '', value).strip().lower() It might be better to do it like this (r'[^\w\s-]', '', ()).strip() Is Python the actual culprit? Yeah it might be. Matt ( urged me to also take a look if Python might be doing this wrong. The İ is the Latin Capital Letter I with Dot Above. It's codepoint is U+0130 According to the chart for the Latin Extended-A set (), it's lowercase version is U+0069. U+0069 lives in the C0 Controls and Basic Latin set (). Lo and behold: it is the Latin small letter I. So a latin lowercase i. Does this really mean that Python is doing something weird here by adding the Combining dot above? Honestly, I can't imagine that and I am probably missing an important thing here because my view is too naive. I hope this shorter explanation makes sense. If it does not, please try to read through the detailed analysis in the repo (). If that also does not make a ton of sense, let me know. In any case, thank you for taking the time to read this bug report. Looking forward to feedback and thoughts. I am happy to oblige in any capacity necessary.


Custom template tags raise TemplateSyntaxError when keyword-only arguments with defaults are provided. Description (last modified by P-Seebauer) When creating simple tags without variable keyword args, but an keyword argument with a default value. It's not possible to supply any other variable. def hello(, greeting='hello'): return f'{greeting} world' {% hello greeting='hi' %} Raises “'hello' received unexpected keyword argument 'greeting'” Also supplying a keyword argument a second time raises the wrong error message: #tag def hi(, greeting): return f'{greeting} world' {% hi greeting='hi' greeting='hello' %} Raises “'hi' received unexpected keyword argument 'greeting'” instead of "'hi' received multiple values for keyword argument 'greeting'" Same goes for inclusion tags (is the same code) I already have a fix ready, will push it after creating the ticket (that I have a ticket# for the commit). Is actually for all versions since the offending line is from 2.0…


Resetting primary key for a child model doesn't work. Description In the attached example code setting the primary key to None does not work (so that the existing object is overwritten on save()). The most important code fragments of the bug example: from import models class Item(models.Model): # uid = models.UUIDField(primarykey=True, default=uuid.uuid4, editable=False) uid = models.AutoField(primarykey=True, editable=False) f = models.BooleanField(default=False) def reset(self): = None self.f = False class Derived(Item): pass class SaveTestCase(TestCase): def setUp(self): self.derived = (f=True) # create the first object item = () obj1 = item.derived () () # the first object is overwritten def testftrue(self): obj = () self.assertTrue(obj.f) Django 2.1.2


FileInput shouldn't display required attribute when initial data exists. Description (last modified by thenewguy) I think that ClearableFileInput.userequiredattribute() () should be moved to FileInput.userequiredattribute() so that required is not output on the html input element that represents FileInput when a file is already set (e.g. already saved on a model instance that is being edited). Maybe I am overlooking a use case where this is not desirable? I can not think of one.


admin.E130 (duplicate name attributes of actions) should specify which were duplicated. Description The fact that the name is used is somewhat an implementation detail, and there's no guarantee the user has enough of an understanding of python to know what that attribute is, let alone how to fix it. This just came up on IRC because a user had defined actions = [deleteselected] where deleteselected was a reference to their own callable, but shares the name of the base one (and by specifying the actions = they were assuming that they were wholesale replacing the actions list, where that may not be true for site-wide actions) so errored ... but they only had define a list of len(...) == 1 so how can there be a duplicate (is their thought process)? The error message should specify those names that occur 2(rather than just check len(...) vs len(set(...))), and ought ideally to explain where the duplicate comes from (ie: AdminSite-wide). Related ticket about E130: (+ those it references) but is about the replacement strategy rather than the error message itself.


Model.getFOOdisplay() does not work correctly with inherited choices. Description (last modified by Mariusz Felisiak) Given a base model with choices A containing 3 tuples Child Model inherits the base model overrides the choices A and adds 2 more tuples getfoodisplay does not work correctly for the new tuples Example: class A(models.Model): foochoice = [("A","output1"),("B","output2")] fieldfoo = models.CharField(maxlength=254,choices=foochoice) class Meta: abstract = True class B(A): foochoice = [("A","output1"),("B","output2"),("C","output3")] fieldfoo = models.CharField(maxlength=254,choices=foochoice) Upon invoking getfieldfoo_display() on instance of B , For value "A" and "B" the output works correctly i.e. returns "output1" / "output2" but for value "C" the method returns "C" and not "output3" which is the expected behaviour


translation.E004 shouldn't be raised on sublanguages when a base language is available. Description According to Django documentation: If a base language is available but the sublanguage specified is not, Django uses the base language. For example, if a user specifies de-at (Austrian German) but Django only has de available, Django uses de. However, when using Django 3.0.2, if my has LANGUAGECODE = "de-at" I get this error message: SystemCheckError: System check identified some issues: ERRORS: ?: (translation.E004) You have provided a value for the LANGUAGECODE setting that is not in the LANGUAGES setting. If using LANGUAGE_CODE = "es-ar" Django works fine (es-ar is one of the translations provided out of the box).


Raise a descriptive error on update()/delete() operations following (), intersection(), and difference(). Description (last modified by Joon Hwan ) bfilter() seems to merge but does not apply to the actual update q = () q = q.union(M.objects.afilter()) print(q) q = q.union(M.objects.bfilter()) print(q) q.update(name='foo') <QuerySet [<M: M object (675)]<QuerySet [<M: M object (675), <M: M object (773)]UPDATE "m" SET "name" = "foo" WHERE ORDER BY U0."id" ASC LIMIT 1); args=(...)


Enumeration Types are not usable in templates. Description (last modified by Mariusz Felisiak) The new enumeration types are great but can't be used in Django templates due to their being callable. For example this doesn't work: {% if student.yearinschool == YearInSchool.FRESHMAN %} This is because YearInSchool, being a class, is callable, and Django Templates always call callables with no arguments. The call fails because the required value argument is missing. The easy solution would be to declare donotcallintemplates = True on the various Choices classes.


Named groups in choices are not properly validated in case of non str typed values. Description In case of using typed choices and string value to store it (in my case it is multiple values stored in char field as JSON) it is possible to catch error while run makemigrations (checkchoices error): main.MultiValueFieldModel.multivaluefieldintegerwithgroupedchoices: (fields.E005) 'choices' must be an iterable containing (actual value, human readable name) tuples. Looking deeper into the django code, we see actual error message: 'int' object is not iterable and it happens in this block of code (): if self.maxlength is not None and groupchoices: choicemaxlength = max( choicemaxlength, (len(value) for value, in groupchoices if isinstance(value, str)), ) If we have CharField (any other with maxlength defined) and grouped choices with non str typed values like: choices=( ('one', ((1, 'One',), (11, 'Eleven',),),), ('two', ((2, 'Two',), (22, 'Twenty two',),),), ) we will have the situation, when max function receives only one integer value (choicemaxlength), because (len(value) for value, in groupchoices if isinstance(value, str)) will return empty generator, and that is why error 'int' object is not iterable raises (max function waits iterable if there is only one argument). Code block: choicemaxlength = max( choicemaxlength, (len(value) for value, in groupchoices if isinstance(value, str)), ) in this case works like: choicemaxlength = max( choicemaxlength, [], ) which is incorrect. The simples solution is to add one additional argument to max function, which will be usefull only in this partucular situation: choicemaxlength = max( choicemaxlength, 0, (len(value) for value, in group_choices if isinstance(value, str)), )


JSONField are not properly displayed in admin when they are readonly. Description JSONField values are displayed as dict when readonly in the admin. For example, {"foo": "bar"} would be displayed as {'foo': 'bar'}, which is not valid JSON. I believe the fix would be to add a special case in to call the prepare_value of the JSONField (not calling directly to take care of the InvalidJSONInput case).


makemigrations does not detect/like model name case changes Description Starting with class Evidence(models.Model): rubrictype = models.ForeignKey('Rubrictype') class Rubrictype(models.Model): typecode = models.CharField(maxlength=1) Make the initial migration: $ makemigrations Migrations for 'asmigrations': Create model Evidence Create model Rubrictype Add field rubrictype to evidence Change the name of Rubrictype to RubricType: class Evidence(models.Model): rubrictype = models.ForeignKey('RubricType') class RubricType(models.Model): typecode = models.CharField(maxlength=1) Generate the migration: $ makemigrations Migrations for 'asmigrations': Alter field rubrictype on evidence Django does not detect the name change on the RubricType model itself. No confirmation is requested for the name change and no operation is generated. The problem is that any subsequent makemigrations run will generate the same operation ad infinitum: $ makemigrations Migrations for 'asmigrations': Alter field rubrictype on evidence If instead the name is changed to RubricXype: class Evidence(models.Model): rubrictype = models.ForeignKey('RubricXype') class RubricXype(models.Model): typecode = models.CharField(maxlength=1) the corresponding migration becomes $ makemigrations Did you rename the asmigrations.Rubrictype model to RubricXype? [y/N] y Migrations for 'asmigrations': Rename model Rubrictype to RubricXype This migration generates a RenameModel operation only and any subsequent makemigrations runs will properly report "No changes detected". So it appears the change detector does not pick up on capitalization changes in model names.


Trying to work around by adding a migrations.RenameModel( oldname='Rubrictype', newname='RubricType', ) to the auto generated operations results in a ValueError exception when makemigrations is run again: $ makemigrations Traceback (most recent call last): File "", line 10, in <moduleexecutefromcommandline() File "/home/svencoenye/developer/djangotest/lib/python2.7/site-", line 385, in executefromcommandline utility.execute() File "/home/svencoenye/developer/djangotest/lib/python2.7/site-", line 377, in execute self.fetchcommand(subcommand).runfromargv() File "/home/svencoenye/developer/djangotest/lib/python2.7/site-", line 288, in runfromargv self.execute(args, ) File "/home/svencoenye/developer/djangotest/lib/python2.7/site-", line 338, in execute output = (args, options) File "/home/svencoenye/developer/djangotest/lib/python2.7/site-", line 111, in handle convertapps=applabels or None, File "/home/svencoenye/developer/djangotest/lib/python2.7/site-", line 42, in changes changes = self.detectchanges(convertapps, graph) File "/home/svencoenye/developer/djangotest/lib/python2.7/site-", line 109, in detectchanges self.oldapps = (ignoreswappable=True) File "/home/svencoenye/developer/djangotest/lib/python2.7/site-", line 89, in render model=lookupmodel, ValueError: Lookup failed for model referenced by field asmigrations.Evidence.rubrictype: asmigrations.RubricType The sequence of the operations does not matter. Neither does substituting the RenameModel operation for the AlterField operation. (Looking at the next to last entry in the traceback, the autodetector seems to be looking for the new name in the oldapps state?) It is possible, however, to go the long way around and use two separate migrations: Rubrictype -RubricXype. RubricXype -RubricType works without getting the migration state stuck and does not throw an exception.


pk setup for MTI to parent get confused by multiple OneToOne references. Description class Document(models.Model): pass class Picking(Document): documentptr = models.OneToOneField(Document, ondelete=models.CASCADE, parentlink=True, relatedname='+') origin = models.OneToOneField(Document, relatedname='picking', ondelete=models.PROTECT) produces Add parentlink=True to class Picking(Document): origin = models.OneToOneField(Document, relatedname='picking', ondelete=models.PROTECT) documentptr = models.OneToOneField(Document, ondelete=models.CASCADE, parentlink=True, relatedname='+') Works First issue is that order seems to matter? Even if ordering is required "by design"(It shouldn't be we have explicit parentlink marker) shouldn't it look from top to bottom like it does with managers and other things?


Admin: Render foreign key models as links for readonly users Description In the admin UI, when viewing a model for which you have view only permission, foreign key / m2m fields are rendered as plaintext representation of the target object. It would be nicer to render those as links instead so that a readonly user can navigate through the relationships. The link should only be rendered if the user has permission to view the target model. django-developers discussion: !topic/django-developers/XFoaohDpqZE


Add system check for the length of auth permissions codenames. Description I stumbled across this while performing some migrations on models with rather... descriptive names (my original model was dynamically created). Anyway, it looks like in cases where a model name is just under the 100 character limit, and is used, the codenames generated for the default permissions (ie. add, change, delete, view) can exceed the maximum 100 characters on the Permission model. As an example, the model below having a 98 character name allows migrations to be generated, but upon running the migration, a database error is raised when it tries to create permissions with codenames above 100 characters. The model: class SomeExcessivelyDescriptiveModelWithAnAbsolutelyRidiculousNameThatCouldntEverNeedToBeNearlyThisLong(models.Model): field1 = models.CharField(maxlength=25) The error: value too long for type character varying(100) While I'm aware that you can override the default permissions by setting Meta.defaultpermissions on the model (this is how I fixed my app), and that the majority of people would never need model names this long, I figured it might be worth adding validation for this case since it uses 's default behaviour. Also, the error message originally came up while using a flush command, which seemed counterintuitive.


Detection of existing total ordering in admin changelist should take into account UniqueConstraints without conditions. Description I've been fiddling with db indexes lately to improve the performance of an admin view. Eventually I found this PR  which ensures the records displayed by ChangeList are deterministically ordered. Among other things, the code looks for the presence of a uniquetogether attribute on the Meta class of the model to determine if the model is already totally ordered. I think that getdeterministicordering() should check for the presence of UniqueConstraints as well, which currently aren’t considered by the discovery algorithm. I noticed the docs currently advise against using unique_together in favor of UniqueConstraint, suggesting the former may be deprecated in the future, so this fix would change that part of the code accordingly. I'm willing to submit a PR for this. Cheers, Fabio


Raising error about protected related objects can crash. Description (last modified by Matthias Kestenholz) ERROR: testprotectvia () Traceback (most recent call last): File "", line 59, in testPartExecutor yield File "", line 605, in run testMethod() File "", line 99, in testprotectvia p.delete() File "", line 941, in delete collector.collect([self], keepparents=keepparents) File "", line 300, in collect error.protectedobjects[0], TypeError: '' object is not subscriptable Pull request follows. (By the way, this came up while testing !msg/django-developers/WmgqJnQ6ioE/b52uACoPAgAJ / the JSONField GSoC pull request. Apart from this crash everything worked fine!)


ModelForm RadioSelect widget for foreign keys should not present a blank option if blank=False on the model Description Unlike the select widget, where a blank option is idiomatic even for required fields, radioselect has an inherent unfilled state that makes the "-------" option look suspiciously like a valid choice. class TestRun(models.Model): datafile = models.ForeignKey(BatchData, ondelete=models.SETNULL, null=True, blank=False) class TestRunForm(ModelForm): class Meta: model = TestRun fields = ['datafile'] widgets = {'datafile': RadioSelect()} renders {{testrunform.datafile}} as <ul id="iddatafile"<li<label for="iddatafile0"<input checked="checked" id="iddatafile0" name="datafile" type="radio" value=""--------- </label</li<li<label for="iddatafile1"<input id="iddatafile1" name="datafile" type="radio" value="1"First Data File </label</li</ulInstead, there should be no checked option for RadioSelect's <inputtags when rendering a new form from a model if blank is not a valid selection.


{% include %} uses gettemplate where it could selecttemplate Description It'd be nice if the Include template tag was sensible enough to allow fallbacks by selecting the most appropriate template, as things like render/rendertoresponse/rendertostring do. It's tripped me up on more than one occasion, and it seems a trivial feature to support, from my limited testing. from django.template import Template, Context tmpl = Template('{% include var %}') ctx = Context({'var':''}) ctx [{'var': ''}] (ctx) ... some HTML output ... ({'var':['', '']}) {'var': ['', '']} (ctx) Traceback (most recent call last): File "<console", line 1, in <moduleFile "", line 140, in render return (context) File "", line 134, in render return (context) File "", line 823, in render bit = self.rendernode(node, context) File "", line 74, in rendernode return (context) File "", line 165, in render template = gettemplate(templatename) File "", line 145, in gettemplate template, origin = findtemplate(templatename) File "", line 138, in findtemplate raise TemplateDoesNotExist(name) TemplateDoesNotExist: ['', ''] The 'fix' is to change this line from gettemplate to selecttemplate, though this might now be slightly complicated by the recent changes in to allow for rendering of Template instances. Changing to selecttemplate on 1.4 yields the results I'd expect: from django.template import Template, Context tmpl = Template('{% include var %}') ctx = Context({'var':['', '']}) (ctx) ... some HTML output ... Both the above shell sessions assume is in INSTALLED_APPS.


Add secure default SECUREREFERRERPOLICY / Referrer-policy header Description the ability for the SECUREREFERRERPOLICY setting to set Referrer-Policy, released in Django 3.0. I propose we change the default for this to "same-origin" to make Django applications leak less information to third party sites. The main risk of breakage here would be linked websites breaking, if they depend on verification through the Referer header. This is a pretty fragile technique since it can be spoofed. Documentation:  The MDN support grid is out of date:


Possible data loss when using caching from async code. Description CacheHandler use instead of , hence it's a chance of data corruption if someone tries to use caching from async code. There is a potential race condition if two coroutines touch the same cache object at exactly the same time.


FileResponse with temporary file closing connection. Description (last modified by Oskar Persson) I think I might've found a regression in . When I run the following tests (in their defined order) against Postgres I get the error below.


import tempfile from import getusermodel from import FileResponse from import TestCase User = getusermodel() class MyTests(TestCase): def setUp(self): = (username='user') def testfirst(self): with tempfile.TemporaryFile() as f: return FileResponse(f) def testsecond(self): pass Running tests... .E ERROR [0.003s]: testsecond (responses.testfileresponse.MyTests) Traceback (most recent call last): File "", line 238, in cursor return self.preparecursor(self.createcursor(name)) File "", line 26, in inner return func(args, kwargs) File "", line 231, in createcursor cursor = () psycopg2.InterfaceError: connection already closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "", line 19, in setUp = (username='user') File "", line 82, in managermethod return getattr(self.getqueryset(), name)(args, kwargs) File "", line 433, in create (forceinsert=True, ) File "", line 66, in save super().save(args, kwargs) File "", line 746, in save forceupdate=forceupdate, updatefields=updatefields) File "", line 784, in savebase forceupdate, using, updatefields, File "", line 887, in savetable results = self.doinsert(cls.basemanager, using, fields, returningfields, raw) File "", line 926, in doinsert using=using, raw=raw, File "", line 82, in managermethod return getattr(self.getqueryset(), name)(args, kwargs) File "", line 1204, in insert return query.getcompiler(using=using).executesql(returningfields) File "", line 1382, in executesql with () as cursor: File "", line 26, in inner return func(args, kwargs) File "", line 260, in cursor return () File "", line 238, in cursor return self.preparecursor(self.createcursor(name)) File "", line 90, in exit raise djexcvalue.withtraceback(traceback) from excvalue File "", line 238, in cursor return self.preparecursor(self.createcursor(name)) File "", line 26, in inner return func(args, kwargs) File "", line 231, in create_cursor cursor = () connection already closed


Calling a form method htmloutput modifies the dict for NONFIELDERRORS if there are hidden field with errors Description Each time the htmloutput method of a form is called, it appends the errors of the hidden field errors to the NONFIELDERRORS (all) entry. This happen for example when the form methods asp() astable() asul() are called multiple time, or any other method that themselves call one of them. For example, a test form with an hidden input field that add errors during the clean call. Python 3.6.5 (default, Apr 25 2018, 14:26:36) Type 'copyright', 'credits' or 'license' for more information IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help. In [1]: import django In [2]: django.version Out[2]: '2.1.7' In [3]: from django import forms ...: In [4]: class TestForm(forms.Form): ...: hiddeninput = forms.CharField(widget=forms.HiddenInput) ...: ...: def clean(self): ...: self.adderror(None, 'Form error') ...: self.adderror('hiddeninput', 'Hidden input error') ...: In [5]: testform = TestForm({}) In [6]: Out[6]: {'hiddeninput': ['This field is required.', 'Hidden input error'], 'all': ['Form error']} In [7]: print(testform.astable()) <tr<td colspan="2"<ul class="errorlist nonfield"<liForm error</li<li(Hidden field hiddeninput) This field is required.</li<li(Hidden field hiddeninput) Hidden input error</li</ul<input type="hidden" name="hiddeninput" id="idhiddeninput"</td</trIn [8]: Out[8]: {'hiddeninput': ['This field is required.', 'Hidden input error'], 'all': ['Form error', '(Hidden field hiddeninput) This field is required.


', '(Hidden field hiddeninput) Hidden input error']} In [9]: print(testform.astable()) <tr<td colspan="2"<ul class="errorlist nonfield"<liForm error</li<li(Hidden field hiddeninput) This field is required.</li<li(Hidden field hiddeninput) Hidden input error</li<li(Hidden field hiddeninput) This field is required.</li<li(Hidden field hiddeninput) Hidden input error</li</ul<input type="hidden" name="hiddeninput" id="idhiddeninput"</td</trIn [10]: Out[10]: {'hiddeninput': ['This field is required.', 'Hidden input error'], 'all': ['Form error', '(Hidden field hiddeninput) This field is required.', '(Hidden field hiddeninput) Hidden input error', '(Hidden field hiddeninput) This field is required.', '(Hidden field hiddeninput) Hidden input error']} In [11]: testform.nonfielderrors() Out[11]: ['Form error', '(Hidden field hiddeninput) This field is required.', '(Hidden field hiddeninput) Hidden input error', '(Hidden field hiddeninput) This field is required.', '(Hidden field hiddeninput) Hidden input error'] This bug affects probably also version 2.2. A simple fix would be to use a copy of the error list before adding the hidden field errors in the file --- 2019-03-17 18:59:04. +0100 +++ 2019-03-17 19:00:08. +0100 -194,7 +194,7 def htmloutput(self, normalrow, errorrow, rowender, helptexthtml, errorsonseparaterow): "Output HTML. Used by astable(), asul(), asp()." toperrors = self.nonfielderrors() # Errors that should be displayed above all fields. toperrors = self.nonfielderrors().copy() # Errors that should be displayed above all fields. output, hidden_fields = [], [] for name, field in ():


fails to restore objects due to ordering constraints Description I hit this problem in a fairly complex projet and haven't had the time to write a minimal reproduction case. I think it can be understood just by inspecting the code so I'm going to describe it while I have it in mind. Setting serializedrollback = True on a TransactionTestCase triggers rollback emulation. In practice, for each database: BaseDatabaseCreation.createtestdb calls connection.testserializedcontents = connection.creation.serializedbtostring() TransactionTestCase.fixturesetup calls connection.creation.deserializedbfromstring(connection.testserializedcontents) (The actual code isn't written that way; it's equivalent but the symmetry is less visible.) serializedbtostring orders models with serializers.sortdependencies and serializes them. The sorting algorithm only deals with natural keys. It doesn't do anything to order models referenced by foreign keys before models containing said foreign keys. That wouldn't be possible in general because circular foreign keys are allowed. deserializedbfromstring deserializes and saves models without wrapping in a transaction. This can result in integrity errors if an instance containing a foreign key is saved before the instance it references. I'm suggesting to fix it as follows: diff --git index .. --- +++ -4,7 +4,7 import time from import apps from import settings from import serializers -from import router +from import router, transaction from import StringIO from import input -128,8 +128,9 class BaseDatabaseCreation(object): the serializedbtostring method. """ data = StringIO(data) for obj in serializers.deserialize("json", data, ): () with (): for obj in serializers.deserialize("json", data, ): () def getdatabasedisplaystr(self, verbosity, databasename): """ Note that loaddata doesn't have this problem because it wraps everything in a transaction: def handle(self, fixturelabels, *options): # ... with (): self.loaddata(fixturelabels) # ... This suggest that the transaction was just forgotten in the implementation of deserializedbfromstring. It should be possible to write a deterministic test for this bug because the order in which serializedbto_string serializes models depends on the app registry, and the app registry uses OrderedDict to store apps and models in a deterministic order.


Serialization dependency sorting disallows circular references unneccesarily. Description The core.serialization.sortdependencies() function takes a list of apps and/or models, and resolves this into a sorted flat list of models, ready to be serialized in that order. This function is intended to make natural foreign keys work, by serializing models referenced by a natural key before the referencing models. When deserializing, this guarantees that natural key references can be resolved, because there are no "forward references". Furthermore, when a circular reference using natural keys is present, this function raises an exception (e.g. "Can't resolve dependencies for someapp.SomeModel in serialized app list") and prevents serialization from completing, since there is no way to guarantee a model ordering that will have no forward references. Note that this ordering is only needed when natural keys are involved, since data is intended to be loaded in a transaction without constraint checks, so numerical foreign keys can be in the wrong order, as long as all referenced data is present at the end of the transaction. This does not work with natural keys, since those are resolved by Python code that needs the referenced objects present in the database to resolve them. However, this sorting is not actually strictly necessary in all cases where it is applied. When circular references are involved, this then actually prevents serialization for no good reason. In particular, this is the case: When running dumpdata without natural keys enabled (which is the default). Even though natural keys might be defined in the models (which causes the sorting and exception), no natural keys will be present in the dumped data, so no ordering is needed. When dumping data intended for loading with loaddata (which I think is the primary usecase for dumpdata?). loaddata will (since 17 months ago in v2.2, see ) automatically handle forward references by deferring setting fields that reference natural keys that are not yet. In this case, sorting is still useful, to prevent forward references where possible, but when there are circular references, it is acceptable to ignore some dependencies rather than preventing serialization from happening alltogether. When serializing data for tests for serializedrollback=True (in ). This is a serialization that does not use natural keys, so no ordering is needed at all.


Note that this serialization happens always (unlike deserialization only happens with serializedrollback=True), so AFAIU this effectively prevents any tests from working on a database with circular references with natural keys defined. The fix for these issues seems to be rather simple: For dumpdata without usenaturalforeignkeys, skip the ordering and just serialize all models in arbitrary order. AFAICS usenaturalprimarykeys is not relevant here, since that only controls omitting the numerical primary key. For dumpdata with usenaturalforeignkeys, do the ordering but do not bail out when there are circular references (instead just ignore some dependencies and produce a best-effort ordering). For test database serialization, also skip the ordering and serialize in arbitrary order. Note that this would remove two of the three calls to sortdependencies() and allow loops in the last remaining instance. This means that sort_dependencies could be modified to allow loops unconditionally, or we could add an argument and default to disallowing loops in case any code outside of django is using this function? Note that is a related, but different issue, concerning the deserialization of data in testcases. I've been working on fixing this and that related issue today and have a basic version working, with testcases (which proved to be quite a challenge, since testing the test subsystem is a bit tricky...). I'll do some additional testing and cleanup and submit a PR soon. Also note that the circular-reference exception was already disabled for self-referencing models in . The fix for that issue simply ignores self-referencing models for sorting, without taking any additional measures to sort instances to prevent problems in deserialization (this code was when the deferred deserialization did not exist yet), so I wonder how much value this exception still has.


DISTINCT with GROUPCONCAT() and multiple expressions raises NotSupportedError on SQLite. Description Contrary to what is suggested in lines 60-64 of , SQLite does support DISTINCT on aggregate functions. One such example is GROUPCONCAT, which is quite similar to PostgreSQL's STRINGAGG. I can't find any canonical links which provide a useful explanation of GROUPCONCAT, but this should be good enough:  I came across this issue when trying to create my own GroupConcat function subclassing Aggregate (similar to the StringAgg implementation from postgres) and noticed that skipping the check in would allow my queries to run as advertised. My code for GroupConcat is: from import Value from import Aggregate class GroupConcat(Aggregate): function = 'GROUPCONCAT' template = '%(function)s(%(distinct)s %(expressions)s)' allowdistinct = True def init(self, expression, delimiter=None, extra): if delimiter is not None: self.allowdistinct = False delimiterexpr = Value(str(delimiter)) super().init(expression, delimiterexpr, extra) else: super().init(expression, extra) def assqlite(self, compiler, connection, extracontext): return super().assql( compiler, connection, function=self.function, template=self.template, **extracontext ) For the record, as the code above suggests, a separate issue is that GROUP_CONCAT only allows DISTINCT when a delimiter isn't specified. After some discussion on #django, an argument was raised in favor of changing the message to say "Django doesn't support...", but I would imagine that skipping the check entirely would simply result in an OperationalError for malformed queries while still allowing users to extend the ORM as needed.


Admin datehierarchy filter by month displays an extra day at timezone boundary. Description (last modified by Lavrenov Ivan) When I authorized by user with not-UTC timezone, like America/LosAngeles , and open filter by date in month, I see one extra day, that follows to the first day of the previous month


Inherited model doesn't correctly order by "-pk" when specified on Parent.Meta.ordering Description Given the following model definition: from import models class Parent(models.Model): class Meta: ordering = ["-pk"] class Child(Parent): pass Querying the Child class results in the following: print(().query) SELECT "myappparent"."id", "myappchild"."parentptrid" FROM "myappchild" INNER JOIN "myappparent" ON ("myappchild"."parentptrid" = "myappparent"."id") ORDER BY "myapp_parent"."id" ASC The query is ordered ASC but I expect the order to be DESC.


fields.E310-E311 should take into account UniqueConstraints without conditions. Description Hello, I'm trying to create migration with this kind of model. class AppUsers(models.Model): name = models.CharField(...) uid = models.CharField(...) source = models.ForeignKey(...) class Meta: constraints = [models.UniqueConstraint(fields=['uid', 'source'], name='appusersuniqueness')] When I start makemigrations command in I've fields.E310  error It says that I should add uniquetogether field in Meta options: (fields.E310) No subset of the fields 'uid', 'source' on model 'AppUsers' is unique. HINT: Add unique=True on any of those fields or add at least a subset of them to a uniquetogether constraint. If I change Meta options to uniquetogether constraint migration passes with no errors. class AppUsers(models.Model): name = models.CharField(...) uid = models.CharField(...) source = models.ForeignKey(...) class Meta: uniquetogether = [['uid', 'source']] As mentioned in docs  unique_together may be deprecated in the future. So I think nobody wants to face this issue when this will be deprecated :) Thanks, Pavel


system checks: admin.E002 could provide a hint but doesn't Description Currently the output is: myapp.MyCustomUserModel: (auth.E002) The field named as the 'USERNAMEFIELD' for a custom user model must not be included in 'REQUIREDFIELDS'. because I accidentally had: USERNAMEFIELD = "email" EMAILFIELD = "email" REQUIREDFIELDS = (USERNAMEFIELD, "full_name",) Ignoring the fact that Django knows it's wrong to have it in there, and could easily just skip it or warn if it's not set, it doesn't make use of the hints available in system checks. I'd like to suggest that a hint could be provided which says which field it's moaning about, something like (open to better wording): HINT: Your username field is currently set to "email", you should remove "email" from your required fields definition. It's a stupidly little thing, but having not had to make a custom user from scratch for a while, my eyes glazed over the not in must not be and I was confused for all of 2 minutes before actually reading it properly.


MultiPartParser support double quotes Description Although the rfc2231 document does not indicate that values can be wrapped in double quotes. However, some third-party tools wrap the value in double quotation marks when wrapping HTTP requests (such as the filename of the file uploaded by PostmanCanary). This results in double quotes for the filename at the end of Django parsing. Postman request body: ---------------------------- Content-Disposition: form-data; name="file"; filename=".txt"; filename*="UTF-8''" Content-Type: text/plain test ------------------------------ django got filename is .txt" not .txt This is not a bug of Django. But I suggest Django do some compatibility processing. CPython is also compatible with this situation.


() incorrectly formats large/tiny floats in scientific notation Description (last modified by Tim Graham) For floats with values larger than 1e16 or smaller than 1e-5, their string representation uses scientific notation in Python, which causes to return an erroneous output. from import format format(0., '.', 2) '9e-19.00' format(1e16, '.', 2, thousandsep=',', grouping=3, forcegrouping=True) '1e,+16.00' This is similar to but that was only fixed for Decimal types.


Child model updates parent model with empty fields making an extra query in multi-inheritance when parent model has custom PK Description While creating a new model object (using multi-inheritance model =Child(Parent)), Django does an extra update query setting parent model fields to empty values. This situation occurs only if we define a custom primary key in a parent model (eg. as an UUID field). An example without custom primary key (correct behavior): class Parent(models.Model): title = models.TextField() class Child(Parent): body = models.TextField() () INTO "appparent" ("title") VALUES ('') RETURNING "appparent"."id" INTO "appchild" ("parentptrid", "body") VALUES (1, '') An example with custom primary key (incorrect behavior): class Parent(models.Model): id = models.UUIDField( primarykey=True, default=uuid.uuid4, editable=False ) title = models.TextField() class Child(Parent): body = models.TextField() () "appparent" SET "title" = '' WHERE "appparent"."id" = '-ae7b-48a6-a2e0-'::uuid INTO "appparent" ("id", "title") VALUES ('-ae7b-48a6-a2e0-'::uuid, '') INTO "appchild" ("parentptrid", "body") VALUES ('-ae7b-48a6-a2e0-'::uuid, '') Python 3.6, PostgreSQL 9.6


Wrong hint about recursive relationship. Description (last modified by Matheus Cunha Motta) When there's more than 2 ForeignKeys in an intermediary model of a m2m field and no through_fields have been set, Django will show an error with the following hint: hint=( 'If you want to create a recursive relationship, ' 'use ForeignKey("%s", symmetrical=False, through="%s").' But 'symmetrical' and 'through' are m2m keyword arguments, not ForeignKey. This was probably a small mistake where the developer thought ManyToManyField but typed ForeignKey instead. And the symmetrical=False is an outdated requirement to recursive relationships with intermediary model to self, not required since 3.0. I'll provide a PR with a proposed correction shortly after. Edit: fixed description.


makemessages doesn't provide feedback when no locale is specified Description (last modified by Cristóbal Mackenzie) makemessages requires that one of three flags be passed to specify locales for message building: --locale to explicitly specify locales, --exclude to specify locales to exclude, or --all to build message files for all locales. When non of these flags are present, the command doesn't show any errors for the user. According to the source code, it should raise CommandError, but that never happens because of a bug in an if statement that checks if a locale has been specified. I've already fixed this in my fork and have submitted a small PR.  Please point out if there are any other necessary steps to move this forward. Thanks!


Logout link should be protected Description There is a logout link in admin app. It is link, not a form. Therefore it is not CSRF-protected. Probably it is not so important to protect logout from CSRF attack, because this fact cannot be used to do anything harmful. So this is just a request for purity. Another reason is that GET request should never change invernal state of the system.


Deprecate providingargs argument from Signal Description The argument is and always has been purely documentational. It provides no functionality or checking. Therefore, these values are stored in memory for no real use. Documentation can be handled just as easily by a code comment or real documentation articles. On a more practical level, I rarely signals, so I recently had to look up their interface. Seeing this providingargs argument required me to think about how to best use it, only to realize it actually goes unused. We can remove this cognitive distraction from the docs. It has caused other small confusion in the past: .


Inconsistent datetime logging from runserver. Description In Django 1.11 and higher, the runserver logging can sometimes be inconsistent. [16/Apr/2018 13:32:35] "GET /some/local/url HTTP/1.1" 200 7927 [2018-04-16 13:32:35,745] - Broken pipe from ('127.0.0.1', ) This is because logging from WSGIRequestHandler uses servertime as calculated using BaseHTTPServer.logdatetimestring. On the other hand, WSGIServer uses logging without providing a servertime. This gets "fixed" in using self.formatTime(record, self.datefmt), which uses a completely different format. Currently we make this at least consistent by providing the datefmt parameter when constructing the logger, but it would be better if they were coded to be in sync (and configurable?). (Looking into it further, it looks like we should be using %(asctime)s instead of %(servertime)s, but would be good if that were the suggested default. In  we see %(server_time)s.)


sqlmigrate doesn't allow inspecting migrations that have been squashed Description This project for another ticket can be used to reproduce:  When running sqlmigrate to pick up migration 0001 in this project, it complains that two migrations have that prefix: $ python sqlmigrate testapp 0001 CommandError: More than one migration matches '0001' in app 'testapp'. Please be more specific. But when trying to be more specific, it's not possible to load it: $ python sqlmigrate testapp 0001initial Traceback (most recent call last): File "", line 21, in <modulemain() File "", line 17, in main executefromcommandline() File "", line 401, in executefromcommandline utility.execute() File "", line 395, in execute self.fetchcommand(subcommand).runfromargv() File "", line 328, in runfromargv self.execute(args, cmdoptions) File "", line 30, in execute return super().execute(args, options) File "", line 369, in execute output = (args, *options) File "", line 64, in handle plan = [([targets[0]], options['backwards'])] KeyError: ('testapp', '0001initial') It would be nice to: A) catch this error and report a nice message as well B) allow inspection of individual migrations that have been involved in a squash. Normally the workflow is to remove the individual migrations some time after committing the squash, but until that is done it could be useful to see their sql.


Subquery annotations are omitted in group by query section if multiple annotation are declared Description (last modified by Johannes Maron) Sadly there is more regression in Django 3.0.2 even after . Background: It's the same query as . I tried upgrading to Django 3.0.2 and now I get duplicate results. Even tho they query should be distinct. Where on 2.2 the queryset yields 490 results, it's 519 on 3.0. A quick diff on the queries still reveals a different grouped by section: This is the new query on 3.0.2: SELECT DISTINCT "campsoffer"."id", "campsoffer"."title", "campsoffer"."slug", "campsoffer"."isactive", "campsoffer"."modified", "campsoffer"."created", "campsoffer"."providerid", "campsoffer"."activitytype", "campsoffer"."description", "campsoffer"."highlights", "campsoffer"."importantinformation", "campsoffer"."minage", "campsoffer"."maxage", "campsoffer"."food", "campsoffer"."video", "campsoffer"."accommodation", "campsoffer"."accommodationtype", "campsoffer"."roomtype", "campsoffer"."roomsizemin", "campsoffer"."roomsizemax", "campsoffer"."externalurl", "campsoffer"."applicationform", "campsoffer"."caseload", "campsoffer"."fieldtrips", MIN(T4."retailprice") AS "minretailprice", (SELECT U0."id" FROM "campsservicepackage" U0 INNER JOIN "campsregion" U2 ON (U0."regionid" = U2."id") WHERE (U0."companyid" = 1 AND U0."option" = "campsoffer"."activitytype" AND STContains(U2."locations", T4."position")) LIMIT 1) AS "inpackage", "campsprovider"."id", "campsprovider"."title", "campsprovider"."slug", "campsprovider"."isactive", "campsprovider"."modified", "campsprovider"."created", "campsprovider".


"logo", "campsprovider"."description", "campsprovider"."video", "campsprovider"."externalurl", "campsprovider"."terms", "campsprovider"."cancellationpolicy", "campsprovider"."privacypolicy", "campsprovider"."applicationform" FROM "campsoffer" LEFT OUTER JOIN "campsbookingoption" ON ("campsoffer"."id" = "campsbookingoption"."offerid") INNER JOIN "campsprovider" ON ("campsoffer"."providerid" = "campsprovider"."id") INNER JOIN "campsbookingoption" T4 ON ("campsoffer"."id" = T4."offerid") WHERE ("campsoffer"."isactive" = True AND "campsprovider"."isactive" = True AND T4."end" = STATEMENTTIMESTAMP() AND T4."isactive" = True AND "campsoffer"."maxage" = 5 AND "campsoffer"."minage" <= 13 AND (SELECT U0."id" FROM "campsservicepackage" U0 INNER JOIN "campsregion" U2 ON (U0."regionid" = U2."id") WHERE (U0."companyid" = 1 AND U0."option" = "campsoffer"."activitytype" AND STContains(U2."locations", T4."position")) LIMIT 1) IS NOT NULL) GROUP BY "campsoffer"."id", T4."position", "campsprovider"."id" ORDER BY "campsoffer"."created" ASC And what it was (and should be) on 2.2.9: SELECT DISTINCT "campsoffer"."id", "campsoffer"."title", "campsoffer"."slug", "campsoffer"."isactive", "campsoffer"."modified", "campsoffer"."created", "campsoffer"."providerid", "campsoffer"."activitytype", "campsoffer"."description", "campsoffer"."highlights", "campsoffer"."importantinformation", "campsoffer"."minage", "campsoffer"."maxage", "campsoffer"."food", "campsoffer"."video", "campsoffer"."accommodation", "campsoffer".


"accommodationtype", "campsoffer"."roomtype", "campsoffer"."roomsizemin", "campsoffer"."roomsizemax", "campsoffer"."externalurl", "campsoffer"."applicationform", "campsoffer"."caseload", "campsoffer"."fieldtrips", MIN(T4."retailprice") AS "minretailprice", (SELECT U0."id" FROM "campsservicepackage" U0 INNER JOIN "campsregion" U2 ON (U0."regionid" = U2."id") WHERE (U0."companyid" = 1 AND U0."option" = ("campsoffer"."activitytype") AND STContains(U2."locations", (T4."position"))) LIMIT 1) AS "inpackage", "campsprovider"."id", "campsprovider"."title", "campsprovider"."slug", "campsprovider"."isactive", "campsprovider"."modified", "campsprovider"."created", "campsprovider"."logo", "campsprovider"."description", "campsprovider"."video", "campsprovider"."externalurl", "campsprovider"."terms", "campsprovider"."cancellationpolicy", "campsprovider"."privacypolicy", "campsprovider"."applicationform" FROM "campsoffer" LEFT OUTER JOIN "campsbookingoption" ON ("campsoffer"."id" = "campsbookingoption"."offerid") INNER JOIN "campsprovider" ON ("campsoffer"."providerid" = "campsprovider"."id") INNER JOIN "campsbookingoption" T4 ON ("campsoffer"."id" = T4."offerid") WHERE ("campsoffer"."isactive" = True AND "campsprovider"."isactive" = True AND T4."end" = (STATEMENTTIMESTAMP()) AND T4."isactive" = True AND (SELECT U0."id" FROM "campsservicepackage" U0 INNER JOIN "campsregion" U2 ON (U0."regionid" = U2."id") WHERE (U0."companyid" = 1 AND U0."option" = ("campsoffer"."activitytype") AND STContains(U2.


"locations", (T4."position"))) LIMIT 1) IS NOT NULL) GROUP BY "campsoffer"."id", (SELECT U0."id" FROM "campsservicepackage" U0 INNER JOIN "campsregion" U2 ON (U0."regionid" = U2."id") WHERE (U0."companyid" = 1 AND U0."option" = ("campsoffer"."activitytype") AND STContains(U2."locations", (T4."position"))) LIMIT 1), "campsprovider"."id" ORDER BY "camps_offer"."created" ASC


forms.ModelMultipleChoiceField should use "invalidlist" as error message key Description The MultipleChoiceField uses "invalidlist", but ModelMultipleChoiceField uses "list" as the key for the similar error message.


Increase default password salt size in BasePasswordHasher. Description (last modified by Jon Moroney) I've made a patch for this here  Which changes the default salt size from ~71 bits to ~131 bits The rational is that modern guidance suggests a 128 bit minimum on salt sizes OWASP:  Python:  NIST:  In the case of NIST this is technically a hard requirement.


Deprecate using getrandomstring without an explicit length Description currently has a default length value (12). I think we should force callers to specify the length value and not count on a default.


Django humanize's intword filter does not accept negative numbers. Description Django's humanize intword filter does not work with negative numbers. I have created a solution using absolute value. Here is my pull request:


Add option to removestalecontenttypes to remove entries for nonexistent apps. Description (last modified by Javier Buzzi) Add an option (disabled by default) to removestalecontenttypes command to remove entries also for nonexistent apps. Based on discussion. PR


Django 3.0: "GROUP BY" clauses error with tricky field annotation Description Let's pretend that we have next model structure with next model's relations: class A(models.Model): bs = models.ManyToManyField('B', relatedname="a", through="AB") class B(models.Model): pass class AB(models.Model): a = models.ForeignKey(A, ondelete=models.CASCADE, relatedname="aba") b = models.ForeignKey(B, ondelete=models.CASCADE, relatedname="abb") status = models.IntegerField() class C(models.Model): a = models.ForeignKey( A, null=True, blank=True, ondelete=models.SETNULL, relatedname="c", verbosename=("a") ) status = models.IntegerField() Let's try to evaluate next query abquery = (a=OuterRef("pk"), b=1) filterconditions = Q(pk=1) | Q(abab=1) query = A.objects.\ filter(filterconditions).\ annotate( status=Subquery(("status")), ccount=Count("c"), ) answer = ("status").annotate(totalcount=Count("status")) print() print(answer) On Django 3.0.4 we have an error column reference "status" is ambiguous and query is next: SELECT (SELECT U0."status" FROM "testappab" U0 WHERE (U0."aid" = "testappa"."id" AND U0."bid" = 1)) AS "status", COUNT((SELECT U0."status" FROM "testappab" U0 WHERE (U0."aid" = "testappa"."id" AND U0."bid" = 1))) AS "totalcount" FROM "testappa" LEFT OUTER JOIN "testappab" ON ("testappa"."id" = "testappab"."aid") LEFT OUTER JOIN "testappc" ON ("testappa"."id" = "testappc"."aid") WHERE ("testappa"."id" = 1 OR "testappab"."bid" = 1) GROUP BY "status" However, Django 2.2.11 processed this query properly with the next query: SELECT (SELECT U0."status" FROM "testappab" U0 WHERE (U0."aid" = ("testappa"."id") AND U0."bid" = 1)) AS "status", COUNT((SELECT U0.


"status" FROM "testappab" U0 WHERE (U0."aid" = ("testappa"."id") AND U0."bid" = 1))) AS "totalcount" FROM "testappa" LEFT OUTER JOIN "testappab" ON ("testappa"."id" = "testappab"."aid") LEFT OUTER JOIN "testappc" ON ("testappa"."id" = "testappc"."aid") WHERE ("testappa"."id" = 1 OR "testappab"."bid" = 1) GROUP BY (SELECT U0."status" FROM "testappab" U0 WHERE (U0."aid" = ("testappa"."id") AND U0."b_id" = 1)) so, the difference in "GROUP BY" clauses (as DB provider uses "", postgresql 11)


Can't replace global admin actions with specialized ones per-admin Description landed in 2.2 (discussion in ), which makes it impossible to replace a generic site-wide action (such as the built-in deleteselected) with a new one. It fails with the admin.E130 system check error. We're seeing this with the qsessions app, which has to delete its session objects in non-bulk mode in order to clear caches:  (For this particular use case, it seems a fix is to instead override modeladmin.deletequeryset within qsessions's SessionAdmin, as that's what the built-in delete_selected action does per  .)


XML serializer doesn't handle JSONFields. Description I have code: data = serializers.serialize("xml", queryset, fields=fields) if I choose specific fields, which are not JSONField, it is ok. But if I choose field, which is JSONField, I receive error File "/Users/ustnv/PycharmProjects/fpgnko/venv/lib/python3.6/site-", line 128, in serialize s.serialize(queryset, **options) File "/Users/ustnv/PycharmProjects/fpgnko/venv/lib/python3.6/site-", line 107, in serialize self.handlefield(obj, field) File "/Users/ustnv/PycharmProjects/fpgnko/venv/lib/python3.6/site-", line 79, in handlefield (field.valuetostring(obj)) File "/Users/ustnv/PycharmProjects/fpgnko/venv/lib/python3.6/site-", line 25, in characters if content and (r'[\x00-\x08\x0B-\x0C\x0E-\x1F]', content): File "", line 182, in search return compile(pattern, flags).search(string) TypeError: expected string or bytes-like object


makepassword shouldn't accept values other than bytes or string as an argument Description (last modified by iamdavidcz) Currently makepassword function accepts almost every Python object as an argument. This is a strange behaviour and it results directly from forcebytes casting objects to str. We should throw the TypeError when passing anything but bytes or str to makepassword. Reasons: programmers unaware of this strange behaviour can accidentally create weak passwords (potential security issue) other libraries raise the TypeError in the same cases (eg. Werkzeug, passlib) it's inconsistent with the documentation that says: It takes one mandatory argument: the password in plain-text. it's inconsistent with validatepassword behaviour (passing anything but bytes or str to validatepassword raises the TypeError with default settings.AUTHPASSWORDVALIDATORS). Discussion: !topic/django-developers/1Ap0zDjFa4E


Add --check flag to migrate. Description (last modified by thenewguy) It would be helpful if there was a flag for migrate that acted similar to makemigrations --check that could be used to stop CI from deploying an application automatically when unapplied migrations exist. This is different from makemigrations --check because the new command flag would tell us if we need to run any migrations where as makemigrations --check tells us if we need to create any migrations. One currently needs to parse output. It seems like this would be universally useful.


Using SimpleLazyObject with a nested subquery annotation fails. Description (last modified by Jordan Ephron) Prior to it was possible to use a SimpleLazyObject in a queryset as demonstrated below. This new behavior appears to be a regression. Models from import User from import models class A(models.Model): pass class B(models.Model): a = models.ForeignKey(A, ondelete=models.CASCADE) class C(models.Model): owner = models.ForeignKey(User, ondelete=models.CASCADE) TestCase from import User from import OuterRef, Subquery from import TestCase from import SimpleLazyObject from ..models import A, B, C class BugTestCase(TestCase): def testbug(self): owneruser = ( (a=OuterRef("pk")) .annotate(owneruser=Subquery(("owner"))) .values("owneruser") ) user = SimpleLazyObject(lambda: User.objects.createuser("testuser")) A.objects.annotate(owneruser=Subquery(owneruser)).filter( owneruser=user ) Sorry for the somewhat arbitrary testcase, hopefully it's sufficient to repro this issue. Results Traceback (most recent call last): File "", line 20, in testbug owneruser=user File "", line 881, in filter return self.filterorexclude(False, args, kwargs) File "", line 899, in filterorexclude (Q(args, kwargs)) File "", line 1297, in addq clause, = (qobject, self.usedaliases) File "", line 1325, in addq splitsubq=splitsubq, simplecol=simplecol, File "", line 1214, in buildfilter condition = self.buildlookup(lookups, reffedexpression, value) File "", line 1123, in buildlookup lookup = lookupclass(lhs, rhs) File "", line 20, in init = self.getpreplookup() File "", line 70, in getpreplookup return () File "", line 968, in getprepvalue return int(value) TypeError: int() argument must be a string, a bytes-like object or a number, not 'SimpleLazyObject'


Add proper field validation to QuerySet.orderby. Description When you annotate a QuerySet with a uuid key, the orderby functionality breaks for the uuid column because the uuid is "not a valid orderby argument". Changing the constant by allowing a "-" from ORDERPATTERN = re.compile(r'\?|[-+]?[.\w]+$') to ORDERPATTERN = re.compile(r'\?|[-+]?[.-\w]+$') fixes this in PostgreSQL. Is there a reason the former pattern was used, is it incompatible with other dbs?


Allow empty message in management command stdout and stderr proxies. Description Django management commands wrap stdout and stderr in an OutputWrapper that adds a \n at the end of the text provided as the out argument. I suggest allowing () and () to add a newline to respectively stdout and stderr. Currently, it fails because msg is a positional argument. PR


Settings are cleaned insufficiently. Description Posting publicly after checking with the rest of the security team. I just ran into a case where () would return several un-cleansed values. Looking at cleansesetting() I realized that we only take care of s but don't take other types of iterables into account but return them as-is. Example: In my I have this: MYSETTING = { "foo": "value", "secret": "value", "token": "value", "something": [ {"foo": "value"}, {"secret": "value"}, {"token": "value"}, ], "else": [ [ {"foo": "value"}, {"secret": "value"}, {"token": "value"}, ], [ {"foo": "value"}, {"secret": "value"}, {"token": "value"}, ], ] } On Django 3.0 and below: import pprint from import getsafesettings (getsafesettings()["MYSETTING"]) {'else': [[{'foo': 'value'}, {'secret': 'value'}, {'token': 'value'}], [{'foo': 'value'}, {'secret': 'value'}, {'token': 'value'}]], 'foo': 'value', 'secret': '', 'something': [{'foo': 'value'}, {'secret': 'value'}, {'token': 'value'}], 'token': ''} On Django 3.1 and up: from import SafeExceptionReporterFilter import pprint (SafeExceptionReporterFilter().getsafesettings()["MYSETTING"]) {'else': [[{'foo': 'value'}, {'secret': 'value'}, {'token': 'value'}], [{'foo': 'value'}, {'secret': 'value'}, {'token': 'value'}]], 'foo': 'value', 'secret': '', 'something': [{'foo': 'value'}, {'secret': 'value'}, {'token': 'value'}], 'token': ''}


Migration crashes deleting an indextogether if there is a uniquetogether on the same fields Description Happens with Django 1.11.10 Steps to reproduce: 1) Create models with 2 fields, add 2 same fields to uniquetogether and to indextogether 2) Delete indextogether -Fail It will fail at , line 378, in deletecomposedindex(), ValueError: Found wrong number (2) of constraints for as this one will find two constraints, the uniq and the idx one. No way to get out of this... The worst in my case is that happened as I wanted to refactor my code to use the "new" (Dj 1.11) Options.indexes feature. I am actually not deleting the index, just the way it is declared in my code. I think there are 2 different points here: 1) The deletion of indextogether should be possible alone or made coherent (migrations side?) with uniquetogether 2) Moving the declaration of an index should not result in an index re-creation


Allow overridding widget in formfieldformanytomany(). Description (last modified by Mariusz Felisiak) It does not work when I set widget param to function formfieldformanytomany(). This is different from the formfieldforforeignkey() function.


Use PostgreSQL TRUNCATE … RESTART IDENTITY keyword to reset sequences in sqlflush() Description Rather than executing an additional query per truncated table, can truncate and reset sequences in a single query by using the RESTART IDENTITY syntax. My project uses the sqlflush() operation internally and profiling shows that it can consume a large percentage of the runtime. Reducing the number of queries to one should help provide a small performance improvement.


Migration doesn't detect precision changes in fields that ManyToMany points to. Description In my case was: class Vulnerability(models.Model): cveid = models.CharField(maxlength=15, primarykey=True) app = models.ManyToManyField(AppVersion) class Meta: managed = True Later, i changed cveid maxlength to 100 and did migration: operations = [ migrations.AlterField( modelname='vulnerability', name='cveid', field=models.CharField(maxlength=100, primarykey=True, serialize=False), ), ] In result: cveid field length was changed, but vulnerabilityid field length in table vulnerabilityapp remain unchanged


Simplify signature of Description The current signature is: def executesqlflush(self, using, sqllist): The using argument can be dropped and inferred by the calling instance: def executesqlflush(self, sqllist): Some internal ises of this method are already doing: (, sql_flush)


QuerySet.Delete - inconsistent result when zero objects deleted Description The result format of the QuerySet.Delete method is a tuple: (X, Y) X - is the total amount of deleted objects (including foreign key deleted objects) Y - is a dictionary specifying counters of deleted objects for each specific model (the key is the of the model and the value is counter of deleted objects of this model). Example: <class 'tuple': (2, {'myapp.FileAccess': 1, 'myapp.File': 1}) When there are zero objects to delete in total - the result is inconsistent: For models with foreign keys - the result will be: <class 'tuple': (0, {}) For "simple" models without foreign key - the result will be: <class 'tuple': (0, {'myapp.BlockLibrary': 0}) I would expect there will be no difference between the two cases: Either both will have the empty dictionary OR both will have dictionary with model-label keys and zero value.


FieldError when migrating field to new model subclass. Description Analogous to . If creating a model subclass and moving a field onto it in the same step, makemigrations works but migrate dies with Local field 'title' in class 'Book' clashes with field of the same name from base class 'Readable'. For example, take this model: from import models class Readable(models.Model): title = models.CharField(maxlength=200) And change to this: from import models class Readable(models.Model): pass class Book(Readable): title = models.CharField(maxlength=200) The migration generates with CreateModel for Book, then RemoveField for But running it produces the error. Reversing the order of the migration operations makes it pass. The auto-detector should be able to use this order.


Allow () for fields with total UniqueConstraints. Description If a field is unique by UniqueConstraint instead of unique=True running inbulk() on that field will fail. Consider: class Article(models.Model): slug = models.CharField(maxlength=255) class Meta: constraints = [ models.UniqueConstraint(fields=["slug"], name="%(applabel)s%(class)sslugunq") ] (fieldname="slug") Traceback (most recent call last): File "", line 90, in runcode exec(code, ) File "<console", line 1, in <moduleFile "/app/venv/lib/python3.8/site-", line 82, in managermethod return getattr(self.getqueryset(), name)(args, *kwargs) File "/app/venv/lib/python3.8/site-", line 680, in inbulk raise ValueError("inbulk()'s fieldname must be a unique field but %r isn't." % fieldname) ValueError: inbulk()'s field_name must be a unique field but 'slug' isn't. It should be pretty simple to fix this and I have a patch if accepted.


Allow makemigrations to skip database consistency checks Description Currently makemigrations always requires an active database connection, due to it executing loader.checkconsistenthistory() here:  Sometimes you are making a change on a machine that doesn't have a database set up or is configured to use a database on a host that isn't resolvable (i.e a postgres host that's only resolvable within docker-compose). If you run makemigrations on such a machine it will fail while attempting to check the migrations consistency, which is quite annoying. The solution seems to either add a backend for such cases or start a database server locally. I'd like to add a flag to skip the consistency check in these situations, or fail gracefully if the connection to the database isn't available.


ManifestFilesMixin.filehash() returning None get's included in hashed filename as 'None'. Description (last modified by Mariusz Felisiak) When returning a string from a custom ManifestFilesMixin.filehash() implementation, the resulting file name is <filepath.<customhash.<extas expected, whereas returning None results in <filepathNone.<ext. Discussion on django-developers supports this behaviour being unintended. Behavior appears to have been introduced with which split the file hashing into a separate method. The following test, when included in the teststorage.TestCollectionManifestStorage test class demonstrates the bug: def testhashednameunchangedwhenfilehashisNone(self): with ('django.contrib.staticfiles.storage.ManifestStaticFilesStorage.filehash', returnvalue=None): self.assertEqual(storage.staticfilesstorage.hashedname(''), '') As suggested by the name of my test, my opinion is that the correct behaviour should be that if filehash returns None, then no hash is inserted into the filename and it therefore remains unchanged. With that in mind, a possible solution is to change the following lines in the hashedname() method (~line 100 in staticfiles.storage): if filehash is not None: filehash = ".%s" % filehash hashedname = (path, "%s%s%s" % (root, filehash, ext)) to if filehash is None: filehash = "" else: filehash = ".%s" % filehash hashedname = (path, "%s%s%s" % (root, file_hash, ext))


Stop minifying only some admin static assets Description Here is a list of JavaScript files in the admin app and their size: 20K 15K 13K 8.8K 7.6K 6.7K 5.9K 5.4K 5.3K 5.2K 3.2K 1.9K 1.5K 1.1K 911 878 674 569 495 379 363 Some things to notice: Only 4 out of 17 files are minified. The largest file, , is only 20 KB. The largest file is also not included in the list of minified files. All uncompressed files are smaller than the size of the 3 font assets, each ~80 KB. All uncompressed files are smaller than the minified jQuery, ~87 KB. I'm not sure if there is a deliberate or historical reason that only a fraction of the static assets are minified, but it looks like it could be an oversight. The approach is inconsistent. Minifying is step a contributor must manually do. The documentation for doing so is here:  This is a step that is easy to forget, myself included. Whether or not one remembers to compress static assets will also affect the outcome of tests. I suggest we drop the minificaiton of admin files altogether. For such small files, it doesn't seem worth it to add a build step and inconsistently at that. In a typical production scenarios, the static assets will be cached and possibly compressed. Compressing static assets largely removes the size gains of minification. Additionally, there are third party apps to fill the role of static asset optimization. I think we should continue to distribute the vendored libraries minified, however as they are not manually handled during typical contributions.


Add an absolutemax parameter to formsetfactory Description The documentation at  seems to indicate (if I understood it correctly) that the purpose of the maxnum parameter is to prevent that someone sends a manipulated, excessively large value in the hidden form field that states the number of (extra) forms that are submitted, whereas it is not (directly) related to the number of forms that are actually POSTed, or initialized via parameter initials. However, following the example at that page, with MyInitials being a list of e.g. 1500 initial values and request.POST containing more than 1500 formsets: ArticleFormSet = formsetfactory(ArticleForm, extra=0) formset1 = ArticleFormSet(initial=MyInitials) formset2 = ArticleFormSet(request.POST) Now, accessing [1000] throws an IndexError exception. The maxnum is at its default value of 1000, but in the above context, it is not expected that formset1 or formset2 is reduced to maxnum forms -- I'd have expected each to have the full number of forms as initialized. Related thread at django-users:


Remove ifequal from the template language. Description No modern project uses ifequal. No one recommends it. I argue it is taking up valuable bytes in the project. Let's remove it.


Deprecate (). Description The docs for  say: This function is an alias to (). It’s likely to be deprecated in a future release. It looks like the change was made in this commit back in 2016 (Django 2.0). Given some years have passed, is it now the time to deprecate this function?


Add check for fields of UniqueConstraints. Description (last modified by Marnanel Thurman) When a model gains a UniqueConstraint, makemigrations doesn't check that the fields named therein actually exist. This is in contrast to the older uniquetogether syntax, which raises models.E012 if the fields don't exist. In the attached demonstration, you'll need to uncomment "withuniquetogether" in in order to show that uniquetogether raises E012.


models.E015 is raised when ordering uses lookups that are not transforms. Description check SystemCheckError: System check identified some issues: ERRORS: app.Stock: (models.E015) 'ordering' refers to the nonexistent field, related field, or lookup 'supplyproductparentisnull'. However this ordering works fine: list(Stock.objects.orderby('supplyproductparentisnull').valueslist('pk', flat=True)[:5]) [1292, 1293, 1300, 1295, 1294] list(Stock.objects.orderby('-supplyproductparentisnull').valueslist('pk', flat=True)[:5]) [108, 109, 110, 23, 107] I believe it was fine until was implemented. is a foreign key to Supply, Supply.product is a foreign key to Product, is a ForeignKey('self', models.CASCADE, null=True)


collectstatic doesn't run staticfiles checks. Description Piecing this together from a user who is having trouble with staticfiles (who doesn't, at least once) on IRC. the system checks framework has a checkfinders which is invoked if django.contrib.staticfiles is in your INSTALLEDAPPS which defers checks to each individually configured finder as () - this accounts for running the following line: if not isinstance(settings.STATICFILESDIRS, (list, tuple)): which is intended to catch the problem scenario of STATICFILESDIRS = () being edited to STATICFILESDIRS = ("mypath") with no trailing comma to make it a tuple, rather than an iterable string. However, the collectstatic management command has requiressystemchecks = False so it appears possible to edit the value and directly run python collectstatic without the type being checked or the error raised. Naively, I'm assuming that something like the following needs to occur: for finder in getfinders(): () # ignoring NotImplementedError # then somehow surface any errors back to stderr as if I've not delved deeply into the "surface errors" part.


Document admin's requirement on django.template.contextprocessors.request context processor. Description Since commit , the admin templates use the implied request variable normally by django.template.contextprocessors.request. As Django templates silence errors, this went unnoticed during testing, and won't immediately break the templates, but certain expected rendering features won't work. Django should document this change: In the release notes (provide a deprecation period where it is a warning only) In the admin docs In the system check framework as a warning (but eventually an error)


Union queryset should raise on distinct(). Description (last modified by Sielc Technologies) After using .annotate() on 2 different querysets and then .union() .distinct() will not affect the queryset def setUp(self) -None: user = self.getorcreateadminuser() (user, name="Sam1") (user, name="Sam2 acid") (user, name="Sam3") (user, name="Sam4 acid") (user, name="Dub") (user, name="Dub") (user, name="Dub") = user def testunionannotateddiffdistinct(self): qs = () qs1 = (name='Dub').annotate(rank=Value(0, IntegerField())) qs2 = (name='Sam1').annotate(rank=Value(1, IntegerField())) qs = (qs2) qs = qs.orderby('name').distinct('name') # THIS DISTINCT DOESN'T WORK self.assertEqual((), 2) expected to get wrapped union SELECT DISTINCT ON () * FROM (SELECT ... UNION SELECT ...) AS siebox_sample


Alias used in aggregate filtering is incorrect. Description With the following queryset: IndicatorValue.objects .values("freight") .annotate( loadingtime=Min("datetime", filter=Q(type=IndicatorValue.TYPEFREIGHTCREATED)) - Max("datetime", filter=Q(type=IndicatorValue.TYPEFREIGHTCOMPLETED)), hastoploading=Exists(OrderItemResult.objects.orderby().filter(orderlineorderfreight=OuterRef("freight"), loadingarmloadingtype=LoadingArm.LOADINGTYPETOP, ).values('pk')), hasbottomloading=Exists(OrderItemResult.objects.orderby().filter(orderlineorderfreight=OuterRef("freight"), loadingarmloadingtype=LoadingArm.LOADINGTYPEBOTTOM, ).values('pk')) ) .aggregate( topmin=Min("loadingtime", filter=Q(hastoploading=True, hasbottomloading=False)) ) I get the following SQL generated for the aggregate (notice that both alias used are the same in the SQL, whereas they are not in the queryset): MIN("loadingtime") FILTER (WHERE ("hastoploading" = false AND "hastoploading" = true)) The full SQL generated is: SELECT MIN("loadingtime") FILTER (WHERE ("hastoploading" = false AND "hastoploading" = true)) FROM (SELECT "indicatorsindicatorvalue"."freightid" AS Col1, (MIN("indicatorsindicatorvalue"."datetime") FILTER (WHERE "indicatorsindicatorvalue"."type" = \'freightcreated\') - MAX("indicatorsindicatorvalue"."datetime") FILTER (WHERE "indicatorsindicatorvalue"."type" = \'freightcompleted\')) AS "loadingtime", EXISTS(SELECT U0."id" FROM "ordersorderitemresult" U0 INNER JOIN "loadingterminalloadingarm" U1 ON (U0."loadingarmid" = U1."id") INNER JOIN "ordersorderitem" U2 ON (U0."orderlineid" = U2."id") INNER JOIN "ordersorder" U3 ON (U2."orderid" = U3."id") WHERE (U1."loadingtype" = \'TOP\' AND U3."freightid" = "indicatorsindicatorvalue"."freightid")) AS "hastoploading", EXISTS(SELECT U0.


"id" FROM "ordersorderitemresult" U0 INNER JOIN "loadingterminalloadingarm" U1 ON (U0."loadingarmid" = U1."id") INNER JOIN "ordersorderitem" U2 ON (U0."orderlineid" = U2."id") INNER JOIN "ordersorder" U3 ON (U2."orderid" = U3."id") WHERE (U1."loadingtype" = \'BOTTOM\' AND U3."freightid" = "indicatorsindicatorvalue"."freightid")) AS "hasbottomloading" FROM "indicatorsindicatorvalue" WHERE "indicatorsindicatorvalue"."deleted" IS NULL GROUP BY "indicatorsindicatorvalue"."freightid", "hastoploading", "hasbottomloading") subquery It works fine with Django 2.2 (which does not use alias there if I'm not mistaken).


Add getresponseasync for ASGIStaticFilesHandler Description It looks like the StaticFilesHandlerMixin is missing the the async response function. Without this, when trying to use the ASGIStaticFilesHandler, this is the traceback: Exception inside application: 'NoneType' object is not callable Traceback (most recent call last): File ".../lib/python3.7/site-", line 30, in asgi await (scope, receive, send) File "", line 86, in call return await super().call(scope, receive, send) File "", line 161, in call response = await self.getresponseasync(request) File "", line 148, in getresponseasync response = await self.middlewarechain(request) TypeError: 'NoneType' object is not callable


Implement autoreload behaviour for cached template loader Description It would be nice to be able get the speed benefit of the cached template loader during development, but without the downside of having to restart the server every time you change a template. It turns out it's possible with just a few changes. Because it's not really possible to configure the cached template loader I did have to base this patch on the fix for . Enabling autoreloading for the cached template loader would work like this: TEMPLATES = [{ 'BACKEND': '', 'DIRS': [(BASEDIR, 'templates')], 'APPDIRS': True 'OPTIONS': { 'cache_templates': True, 'autoreload': DEBUG } }]


Raise CommandError when clearsessions is called on a backend not implementing clearexpired() Description Formal ticket for the issue raised by François Freitag in PR. clearsessions writes to stderr directly when a backend has not implemented clearexpired(). François notes that the recommended behavior is to raise CommandError.


() doesn't work properly for resolved subqueries. Description () doesn't work properly for resolved subqueries, basically all resolved subqueries are now equal. Regression in .


Cannot mix Exists expression with keyword arguments to When Description (last modified by Ryan Heard) I don't seem to be able to provide an Exists expression to When alongside keyword arguments like you can with filter. For instance, consider: class State(models.Model): pass class County(models.Model): name = CharField(maxlength=50) state = ForeignKey(State, relatedname='counties') I can execute the following query just fine: ( Exists((counties=OuterRef('pk'), name="Texas")), name="Dallas", ) But a similar query using When does not work: County.objects.annotate( status=Case( When(Exists((counties=OuterRef('pk'), name="Texas")), name="Dallas", then=Value("DALLAS COUNTY")), default=Value("ELSEWHERE"), )) TypeError: When() supports a Q object, a boolean expression, or lookups as a condition. Instead the arguments must be wrapped in a Q object: County.objects.annotate( status=Case( When(Q(Exists((counties=OuterRef('pk'), name="Texas")), name="Dallas"), then=Value("DALLAS COUNTY")), default=Value("ELSEWHERE"), )) This is inconvenient and inconsistent with how filter works, as shown. When's init method can be modified to allow similar input as filter. Code is in a branch in my repo, but as this is my first time contributing to Django, I want to make sure I open a ticket and get feedback first. Also I wasn't sure how to classify this. I wasn't sure if it was a bug, as I wasn't sure if it was designed this way.


Check that and UniqueConstraint.condition don't span joins. Description (last modified by Mariusz Felisiak) Similar to but for check and condition. Not everyone is familiar with the fact database level constraint cannot span across tables and might be tempted to do class Person(models.Model): age = models.PositiveSmallIntegerField() parent = models.ForeignKey(self) class Meta: constraints = { CheckConstraint( name='ageltparent', check=Q(agelt=parentage) ), } Which we'll happily create migrations for but we'll then crash because we prevent JOINs when resolving check.


Geodjango js template should use for float values to avoid DECIMALSEPARATOR ruin the js syntax Description should use |safe on float values to avoid DECIMALSEPARATOR (and probably other settings in this category) ruin the js syntax by adding unexpected characters instead of dot.


orderby() with expressions crashes on union() querysets. Description (last modified by Mariusz Felisiak) I have read the recent tickets about unions and orderby (, , ) , and my bug is slightly different, so I hope it's not a duplicate. Let's consider two similar models: class EntityA(models.Model): namea = models.CharField(maxlength=128, null=True) dta = models.DateTimeField(null=True) class EntityB(models.Model): nameb = models.CharField(maxlength=128, null=True) dtb = models.DateTimeField(null=True) (namea="a") (namea="qwerty", ()) (nameb="random", ()) (nameb="b") qsa = (name=F("namea"), dt=F("dta")) qsb = (name=F("nameb"), dt=F("dtb")) queryset = (qsb) I can use a simple ORDER BY clause: queryset.orderby("-dt") And everything will work, no problem here. What I actually want is the same query, but with a NULLS LAST Usually the query becomes: queryset.orderby(F("dt").desc(nulls_last=True)) but that raises a DatabaseError: ORDER BY term does not match any column in the result set. I know unions can handle only a few clauses, but ORDER BY is one of them, so I'm unsure whether this is the expected behaviour or not. If it's expected, then the raised exception could be more explicit.


().delete() subquery usage performance regression Description Lock tests are failing with Django-MySQL on Django 3.1:  . The tests run ().delete(). Django 3.0 generates this SQL: DELETE FROM Django 3.1 generates this SQL: DELETE FROM WHERE . IN (SELECT . FROM ) The subquery is a blocker for using LOCK TABLES along with delete() - as per the mysql docs: You cannot refer to a locked table multiple times in a single query using the same name. Use aliases instead, and obtain a separate lock for the table and each alias: mysqlLOCK TABLE t WRITE, t AS t1 READ; mysqlINSERT INTO t SELECT FROM t; ERROR 1100: Table 't' was not locked with LOCK TABLES mysqlINSERT INTO t SELECT FROM t AS t1; Since there's no alias on the subquery, there's no way to lock it. Additionally this change is a performance regression. I benchmarked with MariaDB 10.3 and filling an InnoDB a table of 100k rows via the [sequence storage engine ]. Using the old DELETE FROM, deletion takes 0.2 seconds: root [19]create table t(c int primary key); Query OK, 0 rows affected (0.079 sec) root [16]insert into t select from seq1to100000; Query OK, rows affected (0.252 sec) Records: Duplicates: 0 Warnings: 0 root [17]delete from t; Query OK, rows affected (0.200 sec) Using DELETE FROM WHERE id IN (...), deletion takes 7.5 seconds: root [18]drop table t; Query OK, 0 rows affected (0.008 sec) root [19]create table t(c int primary key); Query OK, 0 rows affected (0.079 sec) root [20]insert into t select from seq1to_100000; Query OK, rows affected (0.594 sec) Records: Duplicates: 0 Warnings: 0 root [21]delete from t where c in (select c from t); Query OK, rows affected (7.543 sec) root [22]drop table t; Query OK, 0 rows affected (0.013 sec) Yes in theory the optimizer should be able to find this, and it may even be fixed in later MySQL/MariaDB versions.


Add hint to the E410 about AuthenticationMiddleware. Description Given an empty MIDDLEWARE in your project, you try and run the admin. The system checks framework kicks in, giving you: (admin.E408) '' must be in MIDDLEWARE in order to use the admin application. (admin.E409) 'django.contrib.messages.middleware.MessageMiddleware' must be in MIDDLEWARE in order to use the admin application. (admin.E410) 'django.contrib.sessions.middleware.SessionMiddleware' must be in MIDDLEWARE in order to use the admin application. You add each of those middlewares to the stack, and run the application: File "", line 23, in processrequest ) % ("CLASSES" if settings.MIDDLEWARE is None else "") AssertionError: The Django authentication middleware requires session middleware to be installed. Edit your MIDDLEWARE setting to insert 'django.contrib.sessions.middleware.SessionMiddleware' before '' Ignoring the fact that it should be a system check, and it's an unfortunate side-effect of spanning 2 separate contrib apps that it isn't, if the order of the errors output was updated, we'd not see this exception in this specific scenario.


Make () strip dashes and underscores Description (last modified by Elinaldo do Nascimento Monteiro) Bug generation slug Example: from import text text.slugify("This is a test ---") output: this-is-a-test- Improvement after correction from import text text.slugify("_This is a test ---") output: this-is-a-test PR


Constant expressions of an ExpressionWrapper object are incorrectly placed at the GROUP BY clause Description I have a function that expects an arbitrary Query expression and constructs a query on a Postgres db def execQuery(expr): expr = ExpressionWrapper(expr, outputfield=IntegerField()) return Model.objects.annotate(exprres=expr).values('exprres', 'columna').annotate(sum=Sum('columnb')) However, when the given expr is a constant expression (e.g., Value(3)), Django generates an SQL query that contains this constant expression in its GROUP BY clause. SELECT "model"."columna", 3 AS "exprres", SUM("model"."columnb") AS "sum" FROM "model" GROUP BY "model"."columna", 3 This leads to an exception because in Postgres, the query above is invalid: aggregate functions are not allowed in GROUP BY LINE 1: SELECT "model"."columna", 3 AS "exprres", SUM("model"."col... Note that when the given query expression is not wrapped by the ExpressionWrapper object, Django correctly identifies and omits the constant from the GROUP BY clause. For example, the query below runs correctly. def execQuery(expr): return Model.objects.annotate(exprres=Value(3, outputfield=IntegerField())).values('exprres', 'columna').annotate(sum=Sum('columnb')) SELECT "model"."columna", 3 AS "exprres", SUM("model"."columnb") AS "sum" FROM "model" GROUP BY "model"."columna"


Memcached key validation raises InvalidCacheKey with clunky message. Description On Django 2.2.13 the code for memcachekeywarnings in has a bad format string that results in raising an exception rather than just producing a warning. This can be reproduced with a memcached key with a space in it, e.g. "foo bar". This code was present before the 2.2.13 release, but becomes more exposed with that release, since it begins validating cache keys. I think it's as simple as removing the , CacheKeyWarning.


DecimalField.topython() raises TypeError on dict values. Description A call to DecimalField.topython() with a dictionary as the value parameter produces TypeError instead of ValidationError. This is a problem, for example, when you try to save a model object, and a decimal field got set to a dictionary by mistake. The TypeError exception that comes back makes it hard to track the problem to the field if the object has a lot of fields. I am proposing a patch to fix it:


Description (last modified by Nicolas Baccelli) I'm migrating my app to django 3.0.7 and I hit a strange behavior using a model class with a field labeled filterable class ProductMetaDataType(models.Model): label = models.CharField(maxlength=255, unique=True, blank=False, null=False) filterable = models.BooleanField(default=False, verbosename=("filterable")) class Meta: applabel = "adminpricing" verbosename = ("product meta data type") verbosenameplural = ("product meta data types") def str(self): return class ProductMetaData(models.Model): id = models.BigAutoField(primarykey=True) product = models.ForeignKey( Produit, null=False, blank=False, ondelete=models.CASCADE ) value = models.TextField(null=False, blank=False) marketplace = models.ForeignKey( Plateforme, null=False, blank=False, ondelete=models.CASCADE ) datecreated = models.DateTimeField(null=True, ) metadatatype = models.ForeignKey( ProductMetaDataType, null=False, blank=False, ondelete=models.CASCADE ) class Meta: applabel = "adminpricing" verbosename = ("product meta data") verbosenameplural = ("product meta datas") Error happened when filtering ProductMetaData with a metadatatype : (value="Dark Vador", metadatatype=self.brandmetadata) Error traceback : Traceback (most recent call last): File "", line 481, in testcheckpolicywarningbyfields for p in ( File "/usr/local/lib/python3.8/site-", line 82, in managermethod return getattr(self.getqueryset(), name)(args, kwargs) File "/usr/local/lib/python3.8/site-", line 904, in filter return self.filterorexclude(False, args, kwargs) File "/usr/local/lib/python3.8/site-", line 923, in filterorexclude (Q(args, *kwargs)) File "/usr/local/lib/python3.8/site-", line 1351, in addq clause, = (qobject, self.usedaliases) File "/usr/local/lib/python3.8/site-", line 1378, in addq childclause, neededinner = self.buildfilter( File "/usr/local/lib/python3.8/site-", line 1264, in buildfilter self.checkfilterable(value) File "/usr/local/lib/python3.8/site-", line 1131, in checkfilterable raise NotSupportedError( ProductMetaDataType is disallowed in the filter clause.


Avoid passing NULL to the IN lookup Description Currently prefetchrelated on a FK passes the NULL through to the database for e.g. authorid IN (NULL, 2). Passing NULL is always unnecessary, since it's not allowed in FK's. There's a small risk from passing NULL that it could lead to incorrect with complex prefetch querysets using PK refs because of NULL's weirdness in SQL. For example with these models: from import models class Author(models.Model): pass class Book(models.Model): author = models.ForeignKey(Author, null=True, ondelete=models.DONOTHING) Prefetching authors on Books, when at least one Book has author=None, uses IN (..., NULL, ...) in the query: In [1]: from import Author, Book In [2]: a1 = () In [3]: (author=a1) Out[3]: <Book: Book object (3)In [4]: (author=None) Out[4]: <Book: Book object (4)In [5]: Book.objects.prefetchrelated('author') Out[5]: <QuerySet [<Book: Book object (3), <Book: Book object (4)]In [6]: from import connection In [7]: connection.queries Out[7]: [{'sql': 'INSERT INTO "coreauthor" ("id") VALUES (NULL)', 'time': '0.001'}, {'sql': 'INSERT INTO "corebook" ("authorid") VALUES (2)', 'time': '0.001'}, {'sql': 'INSERT INTO "corebook" ("authorid") VALUES (NULL)', 'time': '0.001'}, {'sql': 'SELECT "corebook"."id", "corebook"."authorid" FROM "corebook" LIMIT 21', 'time': '0.000'}, {'sql': 'SELECT "coreauthor"."id" FROM "coreauthor" WHERE "coreauthor"."id" IN (NULL, 2)', 'time': '0.000'}] Maybe this could generally be extended to use of _in with non-nullable fields?


Improve makemigrations warning message when calling without an active database connection. Description I was looking at the gis install instructions and I came across an error when running makemigrations. (Error is I had not entered password correctly). However, the error message that is generated is a bit odd, it has a full stop on a new line and shows ( I was also able to get the same error message on a clean project, with a postgres database (not gis) and an incorrect password. I'm not sure if this is a 'bug' but it doesn't look quite right? (gis) PS C:\Users\smith\gis\geodjangopython .\ makemigrations c:\users\smith\pycharmprojects\django2\django\core\management\commands\ RuntimeWarning: Got an error checking a consistent migration history performed for database connection 'default': fe_sendauth: no password supplied . ( No changes detected (gis) PS C:\Users\smith\gis\geodjango


Self referencing foreign key doesn't correctly order by a relation "id" field. Description Initially discovered on 2.2.10 but verified still happens on 3.0.6. Given the following models: class OneModel(models.Model): class Meta: ordering = ("-id",) id = models.BigAutoField(primarykey=True) root = models.ForeignKey("OneModel", ondelete=models.CASCADE, null=True) oneval = models.BigIntegerField(null=True) class TwoModel(models.Model): id = models.BigAutoField(primarykey=True) record = models.ForeignKey(OneModel, ondelete=models.CASCADE) twoval = models.BigIntegerField(null=True) The following queryset gives unexpected results and appears to be an incorrect SQL query: qs = (recordonevalin=[1,2,3]) qs = qs.orderby("recordrootid") print() SELECT "oriontwomodel"."id", "oriontwomodel"."recordid", "oriontwomodel"."twoval" FROM "oriontwomodel" INNER JOIN "oriononemodel" ON ("oriontwomodel"."recordid" = "oriononemodel"."id") LEFT OUTER JOIN "oriononemodel" T3 ON ("oriononemodel"."rootid" = T3."id") WHERE "oriononemodel"."oneval" IN (1, 2, 3) ORDER BY T3."id" DESC The query has an unexpected DESCENDING sort. That appears to come from the default sort order on the OneModel class, but I would expect the orderby() to take prececence. The the query has two JOINS, which is unnecessary. It appears that, since is a foreign key to itself, that is causing it to do the unnecessary extra join. In fact, testing a model where root is a foreign key to a third model doesn't show the problem behavior. Note also that the queryset with orderby("recordroot") gives the exact same SQL. This queryset gives correct results and what looks like a pretty optimal SQL: qs = (recordonevalin=[1,2,3]) qs = qs.orderby("recordrootid") print() SELECT "oriontwomodel"."id", "oriontwomodel"."recordid", "oriontwomodel".


"twoval" FROM "oriontwomodel" INNER JOIN "oriononemodel" ON ("oriontwomodel"."recordid" = "oriononemodel"."id") WHERE "oriononemodel"."oneval" IN (1, 2, 3) ORDER BY "oriononemodel"."rootid" ASC So is this a potential bug or a misunderstanding on my part? Another queryset that works around the issue and gives a reasonable SQL query and expected results: qs = (recordonevalin=[1,2,3]) qs = qs.annotate(rootid=F("recordrootid")) qs = qs.orderby("rootid") print() SELECT "oriontwomodel"."id", "oriontwomodel"."recordid", "oriontwomodel"."twoval" FROM "oriontwomodel" INNER JOIN "oriononemodel" ON ("oriontwomodel"."recordid" = "oriononemodel"."id") WHERE "oriononemodel"."oneval" IN (1, 2, 3) ORDER BY "oriononemodel"."zeroid" ASC ASCENDING sort, and a single INNER JOIN, as I'd expect. That actually works for my use because I need that output column anyway. One final oddity; with the original queryset but the inverted sort orderby(): qs = (recordonevalin=[1,2,3]) qs = qs.orderby("-recordrootid") print() SELECT "oriontwomodel"."id", "oriontwomodel"."recordid", "oriontwomodel"."twoval" FROM "oriontwomodel" INNER JOIN "oriononemodel" ON ("oriontwomodel"."recordid" = "oriononemodel"."id") LEFT OUTER JOIN "oriononemodel" T3 ON ("oriononemodel"."rootid" = T3."id") WHERE "oriononemodel"."oneval" IN (1, 2, 3) ORDER BY T3."id" ASC One gets the query with the two JOINs but an ASCENDING sort order. I was not under the impression that sort orders are somehow relative to the class level sort order, eg: does specifing orderby("-record_rootid") invert the class sort order?


Testing that on a simple case doesn't show that behavior at all. Thanks for any assistance and clarification.


Argon2id should be supported and become the default variety for Argon2PasswordHasher Description (last modified by Si Feng) There were three important changes in the upstream argon2-cffi library since Django 1.10 was released with Argon2 support: (Nov 10, 2016) argon2id support was :  (Apr 9, 2018) argon2id became its default type:  (Aug 18, 2018) its hasher's default memory cost changed from 512 to , and parallelism from 2 to 8, per RFC draft recommendations:  When Django 1.10 was released, only argon2d and argon2i were available, hence the hard-coded argon2i variety in Argon2PasswordHasher. Though Django does not use its hasher, the memorycost = 512 and parallelism = 2 in Argon2PasswordHasher were simply copied from argon2-cffi's hasher back then. Now we should sync Django with upstream updates.


django-admin runserver mostly does not work on Windows Description I frequently encounter the problem that running "django-admin runserver" on Windows fails: <some-venv\scripts\ can't open file '<some-venv\test\Scripts\django-admin': [Errno 2] No such file or directory The command works if run with --noreload. There are probably other conditions that must be met for it to fail, or work. I think it makes a difference, for example, if Django was installed with pip or setuptools. The .exe "console scripts " are different between the two; one type has a bit of readable Python at the end, the other does not. What I do know is that the problem is either caused or enabled by the generated console scripts (the .exe wrappers in venv/Scripts) stripping the .exe extension from their own [0] when passing that to Python to run. If I sabotage this by changing the regex in the file itself (where present), or by adding the extension again in before running the command line, it works fine. To be honest, I have no idea whether this is a bug in Django, or pip, or setuptools, or distutils, or something else entirely. I have spent the last two days trying to figure out what exactly goes wrong, and where, with little success. I'm reporting it here because Django is the only place I could find where it is possible to kludge the code into working. I'll be happy to take my business elsewhere if someone can point the way.


compilemessages needlessly runs msgfmt on unchanged .po files Description I have a project where running django-admin compilemessages takes 1.75 seconds. Running it again, when all the .mo files already exists and are up-to-date, also takes 1.75 seconds. I propose that is changed so that it only invokes msgfmt when it would do anything useful. This can be implemented by checking the mtime of the .po file and the corresponding .mo file. (If statting the .mo file fails, treat that as if the mtime was 0.) Only submit the command to the executor if the mtime of the .po file is greater than that of the .mo file. In effect: don't do anything if the .mo file is newer than the .po file. There is one issue with this: the way the code currently uses the iswritable function. Since it modifies the mtime of the .mo file, you would have to perform the stat of the .mo file before you check if it is writable. (Or, you could just remove the iswritable function and its use. That feature is, in my opinion, of dubious value, and it doesn't appear to be documented.) After I made the changes above, the runtime in the common case where nothing needs to be done was reduced from 1.75 seconds to 0.2 seconds. (Unfortunately, I doubt that I will be able to get a Corporate Contributor License Agreement signed, so I can unfortunately not contribute my change.) 75 seconds may not be much, but when a CI system does it repeatedly, it adds up.