Stupid language question: "all possible" or "any possible" ? All we say about the UTF-8 decoding is: Let unicodesequence be the result of decoding bytearray as a UTF-8 string {{UTF8}}. Fail parsing if decoding fails. But UTF-8 has enough "payload bits" to transfer up to 0x1FFFFF where Unicode is limited to 0x10FFFF. Should we include that in the caution somehow or is that (sufficiently) implicit in "if decoding fails" ? (This is obviously approaching bikeshedding, I'm fine with the current text.)
Changed to 'any'. I think it's implicit in 'if decoding fails'; is there a situation where it wouldn't?
I propose we ban from serialized , in order to avoid a large class of security issues (and to simplify implementations) in C/POSIX environments where is the string terminator. I can see three alternatives here. In the serialization of we add: if byte is %0x00, fail serializing. In the parsing of `Display String we add: if octet is zero, fail parsing. If either end uses a strict UTF-8 encoder or decoder, which do not allow "overlong byte sequences" (see below), this prevents from being transported with DisplayString. Java pioneered a handling of by specifying that it must be UTF-8 serialized as - a so-called "over-long UTF-8 byte sequence". We have presently not specified if the UTF-8 encoding of DisplayString is "strict" where such over-long byte sequences are illegal or "non-strict" where either all or a few over-long byte sequences are accepted. Over-long byte sequences in general are frowned upon, because they can be used to "obfuscate" UTF-8 strings, for instance by encoding a period as 0xc0 0xae to try to escape directory traversal checks. We could change the spec to say that the UTF-8 encoding/decoding must be loose (enough) to handle the Java-way, but I failed to identify a suitable normative reference which didnt bring in a lot of other UniCode/UTF-8 baggage we may not want. We can optimistically serialize any bytes we encounter in the encoded UTF-8 bytearray using the "java-trick" and leave it to the UTF-8 decoder to either reject or accept as it sees fit. In the serialization of we add: If byte is %x00 append "%c0%80" to encodedstring. In the parsing of we add: if octet is zero, fail parsing. I'm a big fan of clear text and simple solutions, and I cannot imagine why anybody would ever try to send through a for non-nefarious purposes, so I am 100% on board with simply banning bytes in the encoded UTF-8 byte-array. But it is a (tiny) loss of generality, and if we want to avoid that, we either make life difficult for implementers in C/POSIX based environments by rejecting this ticket, or we throw our lot with Java's handling of to a greater or lesser degree. Optimistically serializing , as specified in the third alternative is a way to do that without tying ourselves to Java's mast, but it introduces some wiggle-room which, depending on ones point of view, is either desirable (allowing the receiver to reject by using a strict UTF-8 decoder) or undesirable (making it anyone's guess if can be transported in a DisplayString or not.) But Java's handling is a bit of a hack, and allowing (some) over-long UTF-8 byte sequences introduces a class of security issues similar to the one this ticket tries to prevent, so all in all, I think we should just ban in serialized with the first alternative above.
IIUC, Display Strings aren't meant for sending byte arrays, therefore I do not think we need to define a special way for conveying NUL characters. Based on that, IMO the question boils down to if we should ban encoders from sending NUL. I do not have a strong opinion on if that should be forbidden, considering the fact that if the receiving application is suspectible to NUL characters decoders have to be careful in the handling of NUL characters regardless of the encoding requirement. Note that decoders are already expected to treat NUL characters specially if necessary, see Security Considerations. Maybe we can just add a reference from the section describing the decoding logic to the Security Considerations, and call it a day.
One possible good point in favor of banning it is that we know that many configs will just act based on regex pattern search and might find some searched string after a %00 and consider they found what they were looking for, while the implementation will stop before when seeing the zero. I think we should simply say that "0x00" must never be serialized, hence "%00" must never appear in a DisplayString. This implicitly allows end users to write pretty simple protection rules (even regex-based) consisting in blocking the 3-char sequence "%00" in such fields. Encouraging the blocking on the receiver side can be efficient enough to discourage implementations from even trying to purposely send it.
There is a cost to banning it, which is that the Unicode string to DisplayString encoding function is no longer complete. While C strings can't represent embedded NULs, all other languages' string types can. Those languages would then have a harder time writing a DisplayString API, because now encoding can fail. So we're trading off a nuisance for C vs. a nuisance for everyone else. It is unclear to me what the second option buys us. The second option just means there are two ways to spell U+0000. A C decoder that uses a naive percent-decoded in-memory representation would still need to account for somehow. The third option would avoid that, but ultimately it just means the on-the-wire representation of the same U+0000 codepoint becomes more verbose. This, however, suggests a fourth option: keep the abstract data format and wire representation as-is. (All Unicode strings are allowed and is how you say U+0000.) Instead, non-normatively suggest that decoders that use a NUL-terminated in-memory representation MAY account for U+0000 by representing it as in memory.
I'm naively suspecting this could be used to attack some elements: fill such a field with as many %00 as you can and it gets doubled in its final representation (e.g. when passed to a next hop) :-/
By "this", do you mean the in-memory representation? I don't think that's a concern. If passing to a next hop is still over HTTP, the representation you send will continue to be because (under that proposal), that is the one and only representation of U+0000. It would continue to be incorrect to encode it as . If you're worried about a large in-memory representation, I will note that it costs the attacker three bytes ('%' '0' '0') of bandwidth to get you to represent something in two bytes ('\xc0' '\x80'). The attacker would get a better ratio be sending you a bunch of 'a's. But ultimately all this is just a question of what in-memory representation your program uses for U+0000. It just needs to unambiguously represents the possible state space. Indeed I might suggest, rather than doing these hacks, don't use NUL-terminated strings! Then this is moot. I certainly would not go with in anything I write.
I see no reason to single out U+0000 from other control characters. Either allow U+0000, or disallow all control characters. The overlong encoding hack will bring more issues than it will solve.
The only reason to single out U+0000 is that it is the string termination character in C and POSIX environments, and that is still a major cause of CVEs. I would not object to banning all the control characters, but that takes us into UniCode land which is full of dragons... That is a third way to get the indeterminacy of the "java-hack", but it does have the slight benefit that the implementor of the sf-bis parser may know if that over-long UTF8 sequence will work or not. However, if it does not, then nothing is gained with respect to predictability.
I'd prefer to stick with a caution here, rather than a prohibition. That means that endpoints (yes, whatever that means) can make decisions about what they tolerate, but we aren't constraining use. That is, the field can carry anything, but a resource might decide not to accept some values.
How about simply adding: "UniCode category 'Cc' (Control characters) are not permitted, unless the field definition unwisely explicitly permits them."
That's a field-specific parsing rule, which doesn't seem great from an API perspective.
Not if it goes in the "Defining New Structured Fields" section ? I think it is perfectly fair game to insist that fields which accept Control Characters are forced to specify that, and ban them by default. My hope is that adding such text will mean that no field will ever be defined to allow them.
Right, but the software handling that ban isn't a SF processor, it's the field-specific code. I think the most we could do would be something like how we handle field-specific constraint failures in 'defining new...', eg: But that creates a situation whereby the default is that there's nothing in the field's spec about control characters, the SF generic implementation passes them through, and the field-specific code is expected to know that it should reject them. Not great. So a better approach might be:
This is where we realize we have waded into the "no way to be a little bit pregnant with UniCode" sump, isn't it ? I still think that %00 is a valid security concern in C/POSIX environments, but I really do not want to open the UniCode can of worms. Maybe we should just drop this ticket and add a stern security warning that there is no such thing as "safe unicode strings" ?
I'd be good with a general unicode 'there be dragons' security considerations warning.


Implementing Date in Python as a object has an issue -- Python dates are limited to a specific range. Given that it's good to expose SF as idiomatic data structures wherever possible, I poked around a bit for limits on date/time objects in libraries, and found (all expressed as limits on years, as that's the most significant bit): Python - Rust - Go - PHP - JavaScript - Ruby - Java - Should we specify a minimum range of dates that an implementation should support (likely 1-9999) so that developers can be confident of them (as we've done for other data types)?
Hm. Do these differences affect other parts of HTTP as well, such as processing the "Date" field (or fields specific to caching...?).
One would imagine that people put HTTP dates into these structures from time to time, yes.
(it's just that HTTP dates rarely go outside the range 1970 - "forseeable future")
Restricting to a year-based range would IMHO make sense if we actually would be using calendar dates (which we do not).
To be clear, I'm not proposing we restrict it -- just require implementations to support at least a particular range of values, just as we do with many other types. Don't get stuck on years, that's just a rough sizing to get an idea of what we're looking at. I think we'd express it in integer values.
I've commented in the PR that I dont like this, but let me add a bit of context: Handling of time, in real life and in computers, is incredibly messy and one of the most unreadable scientific books of all time is "Calendrical Calculations" which is our best shot at that subject matter. I really do not want us to wade into this tarpit, because everybody who has ever done so, has just added more tar in the process. My really strong preference is if Date was defined as "Seconds since Epoch" according to POSIX" with the full range of signed 15 decimal digits we inherit from Integer, so that we in no way adds any further complexity.
That's effectively what we have, even with this addition. This text is just requiring implementations to support at least a certain range, so that it can be relied upon for interoperability.


OK with me.
Display Strings are a superset of Strings, which are a superset of Tokens. This makes APIs more complex, and makes designers have to think about which structure to use. A couple of thoughts: 1) Should we define best practice, that e.g., you should specify Token or String when you specify String -- similar to HTTP? 2) Should we define a new TokenOrString type to encourage this pattern (both in header definitions and APIs)?
Not sure it would be always a best practice, but there are certainly cases where it is. Thus describing the pattern and making it easier to use it sounds right to me.
I'm probably missing something, but why would we want header designers to use TokenOrString? Given, as you say, string is a superset, if you needed to use a string, why allow tokens too when you can just define the header to simply take string?
For brevity. It happens in legacy header fields all the time (dunno whether that's a good reason, though).
This is because legacy fields didn't actually believe token and string are different types, right? Just that strings were optionally unquoted when you don't need quotes. But guidance on the types is for new fields. And, for better or worse, structured fields went with a data model where quoted and unquoted are different types, rather than alternate encodings of the same types. Now that we've made them different, if you're making a new field that needs to represent string values, use strings. It's true this means designers must pick between two similar types. Perhaps making them different was a mistake. (Though making them the same would mean multiple encodings, which is differently bad.) But we've already done that. It's too late to fully merge them because we'll break everyone who was told structures fields was ready to use and treated them differently. Without fully merging them, generic structured fields APIs are stuck using different types for them. So merging them back needs to be a per-field thing, which means fighting with your SF APIs. Instead, the best simplification we can manage is to discourage the TokenOrString half-measure in new fields and say you should just use one or the other.
To the first half of the bug, yes definitely define best practice! Just if the best practice we pick does not align with the data model we picked, we have done something seriously wrong.
We already have best practice about Display Strings in the spec (in that we steer people away from them if they don't need them). We have a separation between Tokens and Strings because they were visible on the wire in pre-existing headers, and we wanted to be able to retrofit / 'pave the cowpaths'. In some cases, they are interchangeable; in others, they are not. I suspect that if we documented best practice for new fields, we'd say "probably use Strings most of the time" because of the underlying reason I raised this issue for -- APIs have very natural mappings for Strings, but for Tokens you probably need some sort of wrapper object or equivalent strategy, so that you can distinguish them from Strings. I'll put together a PR nudging people that way.
Sounds good! So I guess the decision tree is then: Do you need display strings? If yes, use that and only that. Do you have a legacy reason to use tokens? If use, do whatever your legacy reasons need. Otherwise, just use ASCII strings.


See URL Lots of boring editorial tweaks to ensure better consistency both within the document and other HTTP specs.
One related question: The style guide also mentions that "examples should be in HTTP/1.1 format unless they are specific to another version of the protocol", but the draft uses a notation that resembles HTTP/2. Should we change the example messages to HTTP/1.1?
Thank you very for this tedious work! I only spotted a few minor typos.


Since the last draft version was published, we replaced the header with the header. This requires an increase in the interop version as existing clients will need changes to be compatible with the next draft release.


This nags at me a bit. I think you want to be crystal clear are what parts of Stuctured Fields you are and what you aren't. As far as I see, you want to use the aspect in order to reuse the serialization/deserialization, legal characters, legal range, etc., for the auth parameter values. But you don't need the aspect, which would allow you to parameterise the parameters. Does that sound accurate?
We added this sentence based on a URL from NAME Martin, do you have thoughts here? I think the intent was just to point out that we're in the new structured fields world even though Authorization hasn't been retrofitted yet
In , NAME pointed out that we can't simply use byte sequences from structured fields because authentication parameter values need to be which doesn't allow non-quoted colons. So I switched to double-quotes in dcc520544cf98cf634c9ca0a24119256fd18d58e, but that makes the usage of structured fields somewhat messy. Thoughts?
Copying my comment from the change
Good call. I now added both colons and double-quotes via 690ee072bb450ab631db5b32a23b457cec9bb6be
From Ilari Liusvaara on the : I renamed this issue to track whether we want to keep structured fields or not. Personally, I don't have a strong opinion either way.
I could live without structured fields here, presumably there are servers that already have to handle Authorization fields using the more-traditional encodings
I'll note however that if we go with we'll need to either double-quote it or explicitly ban padding, because is not allowed as part of
While base64 proper is often padded, it is more common for base64url to be unpadded (indeed, I think that the string "base64url" specifically refers to the unpadded version if you follow it to its origin, which I think is URL).
dates back at least as far as and that spec explicitly : None of which is a problem, we just need to be explicit. We know how to do that.
Right, the 7515 thing was what we ended up using in another context, where padding was explicitly not included. Clearly, we shouldn't use it here either[^1]. [^1]: I hold the view that it shouldn't be used in any situation, because if knowing the length is important, you would have a length prefix or delimiter, rendering padding pointless for anything other than protection against traffic analysis. And you can't arbitrarily pad with base64 padding, so that's no good.
I'd ideally want to align with draft-ietf-privacypass-auth-scheme. We reviewed this a lot with NAME and went back and forth and ended up with: So from a syntax standpoint, this is "base64url and you don't need quotes, but do include quotes if you have padding". If you define that you don't use padding, then you don't need quotes. So I think things can be consistent here.
Based on the conversation in the room at IETF 117, I'm inclined to make these no longer structured headers, and to use base64url without padding, without quotes, and without colons.


Based on the discussion in URL, this PR would allow servers to repeatedly generate 104 responses to inform the client about the upload progress. The formulation allows the client to free associated resources (e.g. file buffers) because the server guarantees that the upload offset will never be less than the last reported value. Closes URL
If this is not a mandatory part of the protocol, I wonder how the client would know that the server supports it. It's possible that the client has already forgot about the initial bytes beyond the first ACK when the ACK is received.
I don't think the client must know in advance whether the server will send these additional informational responses. This mechanism is an additional enhancement, but not much should be lost if the server does not implement it. Yes, that can happen, but this can also occur in the current draft version. If the client forgot bytes that come before the reported by the server, it cannot continue the upload. The only save way for the client to regularly forget data is by chunking the upload into multiple requests using and thereby getting a confirmation from the server about the new upload offset. The addition of 104 for progress information does not change much about this approach. If you want to be sure about the offset, you should chunk the upload. But the 104s provide a slight improvement, where the client can already forget data before the request is completed. In a sense, the client should never forget data before the request is completed or a 104 is received if the client wants to recover from upload failures. This PR should just be an enhancement for the client, but not pose any additional issues for it. Does that make sense?
Based on the feedback from and IETF 118, I adjust the text in this PR to clarify what header fields can be present in informational responses: Upload creation: Server may send one or more 104 responses The first 104 response must contain the Location header, subsequent responses must not. The Upload-Offset header is optional Upload appending: Server may send one or more 104 responses The 104 response must not contain the Location header. The Upload-Offset header is optional In theory, this allows a 104 to be generated with no Location or Upload-Offset header, but I don't see a reason against disallowing this. The text now also ensures that the client knows the upload resource URL before it receives a 104 with Upload-Offset. This avoids implementation issues where the client might free upload data before it knows the upload resource URL to resume the upload in cases of failure. Please let me know if these changes improve the draft's readability.
If we are adding strict incrementing requirement to 104 responses, should we also do so for plain offset retrieving?
I would prefer so for consistency.
In a message to the mailing list (URL), Austin Wright (NAME proposed to use informational responses to send information about the upload progress from the server to the client. During the Upload Creation Procedure and Upload Appending Procedure, the server could regularly send an informational response with the latest offset in the Upload-Offset header. The value should reflect the amount of data that has been received and safely stored by the server. This way, the client knows what data has been successfully transmitted and is not needed for resumption anymore. Any resources attached for this data (e.g. memory buffers) can be released then. In contrast, If the client only monitors which,data it has sent, it cannot be sure if the server properly stored it yet. There is a short example of this how communication could look like (note: it uses server-generated upload URLs (URL), but it could also be done using client-generated tokens, of course):
I think doing this is fine, and I agree that it has benefits as pointed out above. General guideline of HTTP is that servers are allowed to send any number of informational responses. There's no reason to not use that. Regarding the status code, I think we can reuse 104.
+1 Kazuho
I think we need some communication between client and server to enable this. The protocol is designed to avoid depending on 1xx status codes. Does the client need to explicitly opt in to this? Can the server opt out? What if the server doesn't support 1xx? If we want to keep this optional without adding a new header field, a possible design is: If the server wants to send periodic updates, it can start by doing so in the initial 104. The client can enable additional buffering behavior based on this information (but it might have already dropped some bytes by the initial 104). This is a server driven solution and the client won't be able to opt out.
Or maybe URL
I'm not fussed on the specific response code value but the 102 seems deprecated, for whatever that means, and the linked wording seems at odds with the system design we want
It's not deprecated, but wasn't carried over to WebDAV-bis due to lack of implementations. It's in the registry, so it's a real HTTP status code. That said, I now agree it doesn't fit well, as 102 is for situations where the client is actually waiting for a response (as opposed to still sending).
I guess that's accurate to the historical record, after a very cursory delve into RFC 2518 and 4918. But I wonder how clear that interpretation is to the broader HTTP world. MDN has a perspective you might want to follow up on. Yeah this is where I'm hitting a hard edge. A registered status code, where the behaviour, the important stuff, is described in an obsoleted RFC doesn't really seem great.
FWIW, I would push back hard on that line of argument. The code is registered, so it's a valid status code. And yes, a spec that wants to use might want to either include an updated definition, or make sure the definition is moved to a separate new document.
I also think that 102 is OK to use here (but maybe a new status code is still warranted, see below). The 102 isn't depreciated as a bad idea or because of another better solution, it was merely omitted from recent updates to the WebDAV specification for lack of implementations. I imagine another specification could take over ownership of the definition/registration. I think the only reason I didn't specifically write 102 in my email is, I've to indicate "upload complete, the server is now performing additional processing" and it may be useful to distinguish this signal from "upload has been partially received and committed to storage". The former is probably closer to the intended usage, so it may still be good to mint a new status code for the latter usage. Alternatively, both cases might be distinguished by the HTTP fields that are returned, and maybe servers want to indicate both at the same time ("50% committed to disk, 20% processed").
I share the view that 102 does not fit very well here. It would be awkward to send an URI to which the missing part of the request should be sent, while indicating that "the server has accepted the complete request" (). If we are to choose from an existing status code, 100-continue might be a better choice, though I would prefer using a new status code considering that we did so in introducing 103. All 1xx codes are interim and therefore kind of implies continuation. We can burn a new status code for a particular case of contiuation, which Resumable Upload is.
I opened a PR for this in URL
I think this would be a good addition to the spec as long as it's not mandatory for the server to send this data. As previously stated in this thread It would help the client to discard in-memory data that is no longer required to resume. Would it make sense to also include the location header in the additional 104 responses? This would make the solution more robust in case one (or several) of the 104 responses get lost and the extra header field would not add any significant amount of extra bytes in each request.
Yes, this would be optional. I am unsure how big the benefit would actually be. This would only help if a 104 is lost without the connection being destroyed, so the HTTP response is still open and a subsequent 104 could be received. Maybe this is possible with HTTP/2 or HTTP/3, but I don't know that. Other than that, repeating Location would be redundant.
1xx response are allowed to be dropped by intermediaries. But I'm not aware of any that would drop some in a request. Typically they just swallow all or pass all.
NAME Is that so? I could be wrong but states that a proxy MUST forward 1xx responses unless the proxy itself requested the generation of the 1xx response.
I suspect what NAME meant to say is that some proxies do drop 1xx responses, whether they're allowed to or not. Also, some HTTP libraries consume all the 1xx responses without exposing them to the caller, even if the library itself doesn't do anything with that particular 1xx code; such a library is indistinguishable from a poorly-behaved intermediary.
My concern isn't so much proxies dropping one or more response as much as e.g. switching network and losing one or more of the 104 response (think spotty Wi-Fi moving over to 4G). Another input here would be consistency, especially if we add the header to the first 104. I agree that the benefit might not be huge but it's not zero either. It does however seem like the consensus in Prague was that this is not needed and I can get behind that.
One or more of the 104 responses could be dropped, along with the final response, due to the connection being interrupted, but selective 104s can't be dropped due to network issues. Clients need to deal with failed connections no matter what, so I don't see how this is an issue.
NAME Right, then this is a non-issue that can be skipped. I just wanted to bring it up so that we do not miss aspects. :)
Thanks for the feedback! I adjusted URL to clarify what header fields can be present in 104 responses.


From the list: At the end of Section 2.1 in URL, there is this: Can that last line be expounded on and clarified? Will a "\" character in a label be 'eventually' encoded as "\" or "%5C%5C" or some combination of them?
I think we can add an example for this


CC NAME comment syntax: URL "Handling Ballot Positions": URL Why is the value defined as a String that can be comma-separated as opposed to a List (of Strings)? Up to you, but feel free to reference RFC 3493 section 6.1 for getaddrinfo and discussion of the AI_CANONNAME flag. "might not available" -> "might not be available"
The nit was fixed with
For the list-of-strings, parameters themselves can only contain bare items, not inner lists. URL


IANA early review highlighted that we were attempting to register to the old table. Lets fix that and do a little more tidyup.
Thank you!


This removes the hashes negotiation and changes the Available-Dictionary request header to explicitly use sha-256 as discussed in URL If sha-256 becomes insufficient, the hash negotiation can then be added (or different mechanisms can be used). This is a rebased version of the (sorry, it got messy so it was cleaner to resubmit a clean one).


Discussion at IETF 118 was that the didn't necessarily need to be short to reduce variations (was originally planning on 7 days) but chaning it to last-fetched instead of last-written is important. This updates the default to 14 days as a balance (can always be overridden explicitly).


Adds a new "Content-Dictionary" with the hash of the dictionary used when encoding the HTTP response. For
NAME this adds the dictionary hash to the HTTP response when used. Right now it is a . Let me know if you think it should be a instead.
Mostly for developer ergonomics to make it easier to deploy. In the case of static resources using previous versions as dictionaries (v1 to v2 of URL), one method for enabling that is to compress the new version using the old version on the cli as part of the push to production and store the dictionary-compressed artifact as a file on the server next to the full file. Then at serving time, compare the value of the request header with the file names on the server for the given URL to see if a delta-compressed artifact is available. Structured fields encode the binary data as base-64 encoded strings which aren't filesystem-friendly so there would need to be some steps at serving time to decode the header, convert it to ascii and then check. It's not insurmountable but it's a bit more processing and a straight string match could be handled directly with templating in server config. Lowercase hex was picked since that's what all of the cli tools for generating hashes emit.
Is a reasonable name for the response header? I'd actually prefer but it felt like that might be over-stepping since there could be some other generic need for associating a dictionary with a HTTP-response but maybe I'm being overly-cautious.
I don't think that authoring convenience outweighs the advantages of having a unified grammar here. It's not like base64 is some esoteric function. If you are talking about sha256sum, here's a toy: URL
I agree with NAME - don't. To make the sf value filesystem friendly, you just need to strip a few characters and remap "/".
Keep in mind that hashes are not hex strings. A SHA-256 hash is 32 bytes of binary data. The hex thing is just what some tools happen to output. But anything consuming this programmatically to, e.g., compare the hash against something would be best served by the underlying bytes. Using base64 is also slightly more compact on the wire. And if binary structured fields gets off the ground, it can be even more compact.
My main concern is trading off a unified grammer of using sf-binary in the headers vs consistency with tooling and how the dictionaries are represented once it gets out of the protocol. In the dynamic case, this could be something like the web server checking to see if the requested dictionary is available (on disk) in response to a request. I don't know how each web server will choose to build that but having files named with the hash is an easy common starting point. Otherwise each chooses what the ascii representation of the 32-byte hash should be for their case. On the static case, I'm not so much worried about not having a tool that can create base64-encoded strings of sha256 hashes (it's trivial as you pointed out). I'm more concerned about the transformations that need to happen at serving time to decode the hash, convert it into an ascii representation and then check if that variant of the encoded file is available. It's certainly all doable, but it requires base64decode and binhex (or some equivalents) to be available in the layer where the checks are being done - or to rely on application code. Specifically, I was thinking that it would be easy enough in VCL or an existing Nginx or Apache config to do something like: If request header is present and validates against a [0-9a-f]+ regex and 'br-d' is available in the request header try returning + '.' + request['Accept-Encoding'] + '.br-d' (and add relevant response headers) otherwise return Doing similar processing with base64-encoded hashes has a step to convert it into an unspecified format that is platform-specific and may not be doable without either running application code or updating the server with base64decode + binhex support. It likely doesn't matter for the case of applications and servers that are dictionary-aware and can operate on the raw hashes (and enjoy the ~20 byte request header savings) but it is a tradeoff that pushes complexity further up the stack so I just want to make sure that is the tradeoff that we want to make.
Having a finally consistent grammar for HTTP is the precursor to get consistent tooling. It may be more convenient to do something else in the short term, but in the long term we'd just be adding to the mess of inconsistency in HTTP. In the long term, I would hope that these config language DSLs would be able to process structured fields natively. (Although, it may be on this WG to demonstrate this being possible, since the data model for structured fields is slightly interesting...) I also don't think we should be encouraging, much less designing for, folks trying to process HTTP headers with regexps. There's a well-defined grammar and inventing other parsers is a great way to not quite get things right. We already have a pretty hard time with HTTP extensibility due to folks parsing it wrong. The path-unfriendliness is a little unfortunate. Possibly we should have picked the URL-safe version of base64, but too late for that. But since byte strings and text strings aren't the same type, I imagine any sf-binary-aware config language DSL would also have hex and base64 functions available. Indeed taking attacker-supplied input and mapping it to a path component, without running it through some very restrictive encoding, seems a pretty bad idea! At the end of the day, SHA-256 hashes are byte strings, and sf-binary is the way HTTP decided to say byte string. We could say that, although SHA-256 hashes are byte strings, dictionary names are ASCII strings computed by hex(sha256). Then these headers should be sf-string (with quotes). But that would be less efficient on the wire, and our on the web is to put user needs (bandwidth, in this case) first, so I think this is the wrong tradeoff. That said, I see also uses this format. It wouldn't make sense for and to use different formats, so perhaps switching away from the less consistent and less efficient format should probably be handled separately from this PR?
I switched and both to sf-binary. The plan is to update the ID and then send out a summary of changes for discussion on the listserv and since this PR is about the dictionary in the headers anyway it should be fine to update both of them at the same time. Happy to split it out if it helps with history tracking.
I think this change adds significant complexity, and I'm not sure regarding its benefits. I very recently played around with prototyping a compression dictionaries deployment and was very happy with the simplicity of it: You generate diff files at build time with some naming convention Then at serving time, you configure your routing logic (similarly to how NAME ) to those diffs That can be done today in almost any deployed routing layer. Once this change lands in implementations, a similar deployment would require me to add custom logic to transform that header value to something that is file system friendly. It doesn't matter if that transformation is to base64 decode the binary data and then to hex encode it, or to 'just.. strip a few characters and remap "/"'. It is an operation that is not necessarily supported in many layers that developers currently operate in. To take just one example - if I were to implement this as a , I'm not even sure it's feasible to do that. (although it might be, with a creative ). Can you elaborate on the advantages of moving to base64, beyond theoretical purity?
I don't understand how a client determines which dictionary was used by the server. It seems like the response depends on the request, which is fragile (and doesn't allow clients to advertise multiple dictionaries, which seems likely with Use-As-Dictionary as it is). cc NAME Moved from private ID
For identifying the dictionary, it's worth noting that the spec requires that the client advertise a SINGLE dictionary that it supports but yes, the response depends on the dictionary identified in the request. This also helps with minimizing the variants that caches would need to store since the response needs to be varied on the request's available dictionary (and multiple would explode the permutations). It wouldn't hurt to echo the dictionary hash in the response which would allow for other mechanisms of advertising dictionaries but that also carries the cost of complicating the vary logic (would need to vary based on whatever request header was used to negotiate the dictionary that ended up being used).
Dictionary hash used is now in the response headers.
If this field carries binary data, why aren't you using structured fields?


As brought up in , resumable uploads may interfere with intermediaries scanning the request content. This PR adds a paragraph in the security considerations. Closes URL
Some security measures that involve looking at incoming HTTP requests might be circumvented by resumable uploads. For example, a filter which looks for certain HTTP headers and corresponding text in the request payload might not be triggered when the header occurs in one request and the matching body fragment in a subsequent request. I don't believe there's anything we need to change about the protocol here, but it's probably mentioning this in Security Considerations. Perhaps guidance would be to ensure any security measures are either resumable-upload-aware or are explicitly called to inspect the final request between receipt of the last chunk and execution of the original request.
Thank you, that is a good point. I have opened to add a paragraph in this regard. Please let me know if that expresses the problem correctly.


This adds support for a server-provided sf-string "id" of up to 1024 characters that is echoed back in a "Dictionary-ID" request header when the dictionary is advertised as being available. This allows for a server to use something other than the hash to retrieve the dictionary (i.e. a cache key). This should be more flexible than the client sending the original request URL and will allow for servers to store arbitrary context information with the dictionary. It doesn't open up any additional privacy or security concerns because the dictionary hash was already being treated as if it was user-identifiable information. For
NAME could you please take a look and see if this meets your needs and looks reasonable?
Yes, looks great! Thanks!


This changes the URLPattern match to use a single match pattern to be consistent with other uses of URLPattern.
NAME NAME this is an attempt at a change to have it use a single string for pattern construction. I don't feel great about the same-origin check part of the validation but I don't see why it wouldn't work. It still allows for people to waste bytes by not using path-relative patterns so I may have to re-open the issue around that but should provide the consistency you were asking for. This would be instead of PR .
a couple nitpicks while passing by


After looking harder, I found that RFC 9110 that a proxy "is usually identified by an "http" or "https" URI". I'm not sure that's 100% accurate, but it conveniently resolves the wording problem, and who am I to argue with RFC 9110?
At the moment, section 1.2 (Problems) start with the following sentence: I totally get what it tries to say, but it is confusing if not incorrect. states that a CONNECT request is handled by a "proxy" and the target specified by the "host:port" tuple is the "origin." Therefore, HTTP CONNECT proxies are not identified by an origin. Rather, they are identified by the user's proxy settings. I think we'd better update the problem statement, and might also consider pointing out that the definition of "origin" will be different between CONNECT and connect-tcp (or use different terms to identify the CONNECT target and the entity that creates a tunnel in order to avoid confusion, maybe?).
PS. Practical example showcasing the side effect of the change to the definition is that the way to authenticate becomes different; when a client tries to establish a CONNECT tunnel with authentication, it uses "proxy-authorization" header, whereas when a client tries to establish a connect-tcp tunnel, it uses "authorization." I think it's worth noting.
I'm realizing that this spec has the opportunity to fix the origin issues present in regular CONNECT. While we're there it'd be great to clarify how connect-tcp interacts with Alt-Svc because it's unclear for CONNECT. I sent some more about why it's unclear for CONNECT.
Thanks for the discussion!


Related to and HTTP request proxies use "Proxy-Authenticate" in order to distinguish proxy authentication from destination authentication ("WWW-Authenticate") in the mixed bag of headers that is presented to the proxy. Classic HTTP CONNECT proxies don't have that problem, but they still use "Proxy-Authenticate", probably just for consistency. The MASQUE RFCs don't say anything about which authentication headers to use. "connect-tcp" and MASQUE should definitely follow the same rules, whatever they are. NAME noted that RFC 9110 says "the Proxy-Authenticate header field applies only to the next outbound client on the response chain". This implies that a general-purpose gateway is not expected to forward Proxy-Authenticate headers. However, there is actually no such thing as a general-purpose gateway for Extended CONNECT, so this seems somewhat moot. Personally, I think that our guidance should probably also line up with what we want to say for which is (in my view, despite some authors' protestations) a modernized HTTP request proxy.
An interesting related question is whether MASQUEish resources can be authenticated with other methods, like cookies and bearer tokens. If so, I think they are essentially "content-like" and should probably use WWW-Authenticate. If not, they are essentially distinct, and could conveniently be marked as such with "Proxy-Authenticate". Or perhaps they are both, depending on the context...
I somewhat think it's both — it's content and the proxy.
NAME Thank you for opening the issue. If we are to consider extended CONNECT gateways as proxies, then the question would be what the URI of the connect-tcp request means. RFC 9110 says that target URI specifies the origin. It is only because all the existing extended CONNECT family deals with origins that the TLS handshakes to those servers implementing the extended CONNECT family can be validated using certificates issued for that origin. To conclude, to me in seems that the use of WWW-Authenticate is mandated by the using URI as the target. PS. Note also that RFC 8441 specifically explains how proxies can be used between the client and the extended CONNECT target, see .
I don't think anything is really mandated. Within the current language of RFC 9110, I think we are free to choose either interpretation. However, I'm happy to say that "WWW-Authenticate" is the winner here, and "Proxy-Authenticate" does not traverse "connect-tcp" (or "connect-udp") gateways.
Looks good! :+1:


This is kinda silly, but it seems like adding support for coalescing is something we could fix later. with the nuclear option.
the pull request is fine by me. I do wonder if we shouldn't make the json a tad more complicated to allow for a reasonable mixed definition in the future. But an unreasonable one is obviously possible :)
Can we coalesce http and https? The implication from everything we've said is "yes", but Erik remains reluctant.


This removes the dictionary-specific ttl and switches to using the dictionary resource freshness as the mechanism for expiring dictionaries.
NAME could you please take a look and see if this looks ok? NAME could you look at the language around the freshness of the resource relative to cache-control and stale-while-revalidate and make sure I expressed it properly (and it is something we're ok with leveraging)?
Why does the dictionary lifetime need to be independent of the cache lifetime? Having an object that is cached with a separate lifetime seems unnecessary. I would have said that clients can cache a dictionary for as long as they want (as always), but no longer than the lifetime stated in the cache-control field. Now, you might say that a server might want a resource to be usable in its own right (i.e., as an ordinary resource) for one duration, with a separate lifetime for its use as a dictionary. I see two reasons that this isn't much of a compelling argument: A server can forget about a dictionary and not provide delta compression at any time, so there is no real need for the dictionary lifetime to be shorter than the cache lifetime. Client caches won't want to retain something for longer than its validity, so I don't see much chance that a resource will be kept past its cache lifetime. Resources like this are likely exclusively used as a dictionary. There isn't much value for those resources having a separate lifetime in that case.
The main reason for a separate lifetime for the dictionaries is to allow for sites to reduce the possible variations that are in the wild and reduce the number of variations of that come in and constrain the number of previous releases that they need to compress artifacts for. Otherwise, a dictionary "miss" will result in a full version of the resource getting added to caches for every possible version of where delta-compressed artifacts aren't available. Caches can probably manage that OK on their own, but by reducing the dictionaries to a known window it both provides a lever of control on the variations and also allows for clients to "hit" on a cached version of the resource when they have an old dictionary.
This has been rattling around in the back of my head for the last month or so and I wanted to get my thoughts down for discussion. I think it might be possible to use the existing validity of max-age and stale-while-revalidate to achieve the same result but I'm a bit worried about the complexity that would bring to users. There are 2 main scenarios that express the tension: Versioned immutable resources that change the URL when the content changes. Well-known URLs that are cacheable but update the content in-place (libraries for analytics, ads, embeds, etc). For the purposes of this example, lets assume that the resources are updated twice a day and that 80% of repeat traffic re-visits within 10 days of their previous visit. Assuming a serving architecture with a server-side cache (CDN, load balancer, etc): In both cases, the response is cacheable and keyed based on the dictionary that the client sends as available (). The main concern is how to tune the requests so that the cache behaves like it does today, minimizing traffic to the origin. The "current" version of the resource will be the only one actively fetched by the application independent of when the user last visited. The current best-practice is to cache them for a long time (1 year) and mark them as immutable. Without dictionaries that means that the cache will effectively only see requests for the current version of the resource and all previous versions can age out of the cache (even if they have a long time-to-live, just by virtue of never being accessed). There will be one request to the origin (or a small number) while the cache gets populated but all future requests will always be served from the edge cache. With dictionaries, the cache key depends on when the client last visited the page since the dictionary will be the version of the resource they have in their cache. That means 80% of the requests will be covered by 21 cache keys (one without a dictionary and 20 delta-compressed variants) but that there will be 700+ variants in the wild if the 1-year resource expiration is reused for the dictionary. The cache could manage access frequency to store just the 21 "hot" variants for the last 10 days but the 20% of visits that are using older dictionaries will result in cache misses that require going back to the origin. If a developer shortens the resource cache lifetime to a shorter time, like 5 days, knowing their release schedule then the problem largely goes away. That would require tying the dictionary use to the expiration of the resource and not allow for it to be used even if it is still in cache (but expired). That would mean we are now imposing a short cache lifetime on an immutable resource in order to optimize the hit rate of the middle caches. It feels like explicitly managing the dictionary lifetime at the same place where the dictionary details are specified might be cleaner. For well-known URLs that are not immutable and change the content in-place, we want to be able to use the previous version(s) of a resource as a dictionary for the "current" version. To be able to do that we need the dictionary to be viable beyond the expiration time of the resource. Since we need to hard-clip the use of the dictionaries to prevent cache key issues we'd need to have the dictionaries use something like the time window to allow for a revalidation request of an expired resource while still using the dictionary but allowing for a hard-clip of the dictionary validity at the end of the stale-while-revalidate time. It's theoretically possible but it feels like we're tying the dictionary TTL into the cache heuristics in ways that might not be intended and might also be easy to get wrong.
"well-known" is a term of art you might want to avoid here. I get your intent. It looks like there is a fairly good suggestion for . It might take a bit of time for a cache to be able to work out the length of time from the last access that represents a good time horizon for use. But that seems manageable. The resources that churn is an interesting one. It seems like you are assuming that the cache lifetime is short, such that a replacement version is requested AFTER the cache lifetime expires. Your option does seem like a possible way of managing this, but I'd be unsure whether stale resources are good for use. You'd have to specify that behavior (reasonable, but it would need to be specified). It might be that a longer cache lifetime also works well enough. Clients do tend to re-check earlier than expiration in many cases, which would ensure that the resource is fresh in use, while making the overlap work well.
NAME NAME does using the resource max-age + stale-while-revalidate as and expiration for the client-side dictionary work ok for you (with a recommendation that immutable resources specify expirations based on their typical release cycle)?
I think that having an explicit life time for the dictionary is a little bit more flexible than relying on cache-control headers, but I think that by itself should outlive most dictionary uses I can think of, especially given that the CDN is unlikely to keep those resources in its own cache beyond that period as well.
NAME I'm not sure I understand (or we're not talking about the same part of the issue). The main concern for caches is how long clients consider a dictionary valid for which will dictate how long of a window a given resource will see request headers from. e.g. for a dictionary with a max-age of 1 year, clients will advertise that dictionary as being available for up to a year. If there are 1,000 updates to the resource used as a dictionary (not impossible for a js app that has 3 releases per day) then there will be 1,000 variants of the dictionary in the wild at any given time (with more frequent requests for more recent dictionaries but still some infrequent long-tail requests). To prevent cache misses, it would be beneficial to cap the dictionary validity to a smaller window where most return traffic falls in and have older clients download the full resource from cache rather than missing and going back to the origin. It also caps how many delta-compressed versions need to be created. The discussion point is if that "client valid" time for a dictionary should be its own separate value or if sites should adjust their max-age to be more reasonable instead if they want to limit the variants (capping the validity to the max-age but requiring a shorter max-age rather than blindly setting everything to 1-year).
Worst-case, a separate ttl could be added later if using the resource expiration ends up being a problem blocking adoption.
It seems like there are two different mental models of how caching and cache-misses could work. The difference between them may be contributing to the confusion. The first is that the CDN is keeping "resource A compressed against dictionary X" as a cache item, and if a client makes a request for "resource A compressed against dictionary Z", the CDN needs to go back to origin to retrieve that combination. The second is that the CDN is keeping "resource A" and "resource Z" as cache items, and if a client makes a request for "resource A compressed against dictionary Z", the CDN either has Z locally or not. If not, it will compress resource A in a different way the client supports. (Of course, this doesn't prevent the CDN from having some amount of second-layer caching to avoid recompressing frequent combinations.) Personally, I tend to think the second tends to have better properties -- you don't have an explosion of cache entries or origin requests for the long tail, you simply fall back to a non-dictionary compression scheme. CDNs might choose to count usage as a dictionary as a cache hit for Z, making it stay in cache as long as it continues to be used by clients, but might not choose to fetch Z again if it ages out of the cache. (The interesting quirk from a CDN perspective is that a single resource might have multiple old versions still getting "hits.") The two then probably imply different ways of thinking about lifetimes. In the first case, you could want a separate lifetime on the dictionary to feed into how long the CDN needs to keep around the f(A,Z) item. In the second, the lifetimes of the individual objects likely suffice. (For that matter, if the client is identifying the version of Z it has, then it doesn't really matter whether that object is fresh or not, only whether the server still possesses the indicated version.)
The main use case for this is for something like a web app at build time where they pre-compress the resources at build time for some number of previous releases. At serving time, the CDN doesn't have any special dictionary logic and just honors the headers to vary on and the origin checks for pre-compressed assets using the requested dictionary (in my apps that I was doing this with, I just added the hash to the end of the file name and did a quick file check). On a miss, the origin sends the full resource which ends up in a separate cache key on the CDN. This allows for origins to support dictionary-compressing the content without their CDN or load balancer having to have implemented it.
Right, that's a reasonable workaround for CDNs that don't know about this spec. But wouldn't we rather the CDNs could implement the spec and calculate the delta at the edge? Wouldn't that be the desired end state? Perhaps I'm wrong, or perhaps I've wandered into an architectural question bigger than this issue.
I'm not 100% sure it's just a workaround and origin-compressing against previous releases at build time may be preferable for some cases. The most extreme case that comes to mind is something like a huge WASM app (like Photoshop) where the files are upwards of 50MB (well, huge relative to the usual web page resources). Presumably the CDN could still do the work to do the compression (maybe as an offline job for future requests) so it doesn't require the origin to do it necessarily but it's a lot more reasonable in the extreme cases. That said, I think we can still get buy without a separate TTL and adding one later if it ends up being useful can be done as an enhancement without breaking the existing case.
I like red patches. Thanks.


I guess that's the best we can do, considering that Unicode is a minefield. :-/
... in a swamp.
Is it worth dealing with the fact that 3629 doesn't actually define how to encode a sequence of characters, but we're calling it with one? We'd need to add something to this text, like "for each character (or code point?) in inputsequence, append the result of applying UTF-8 encoding to character to bytearray".
currently does not consider what happens when UTF8 serialization fails, although this could happen with surrogate points. I believe it should explicitly say that. Alternatively, the introduction of display strings could say that only unicode scalars (URL) are allowed (but even then I'd add the extra precaution in the serializer). Finally, we may want to add an example because all of this is far from obvious.
NAME and NAME pointed out that URL doesn't define the and operations that this draft uses. RFC 3629 only defines a character at a time. RFC 3629 also doesn't say to fail encoding or decoding in any particular situations. It says "The definition of UTF-8 prohibits encoding character numbers between U+D800 and U+DFFF", but its definition of encoding doesn't actually prohibit those characters. It also says "MUST protect against decoding invalid sequences", but the definition of decoding doesn't fail in that case. URL is more explicit on these questions and might therefore be a better reference.
Note that the recommended entry points for other specs are URL which are clear about how to handle BOMs as well.
Seems reasonable - will try to work up a PR. I'm likely to keep the general reference to 3629 for UTF-8, and use encoding for the specifics.
I would very very very much prefer not to have a normative reference on a "living" document here. Let's just add the missing needed on top of RFC 3629 to this spec. (and yes, I can make a concrete proposal)
We were clear how to handle BOMs, but what you suggest is a change on top of that.
See minimal PR.
(Posted on the PR by accident:) Is it worth dealing with the fact that 3629 doesn't actually define how to encode a sequence of characters, but we're calling it with one? We'd need to add something to 4.1.11, like "for each character (or code point?) in inputsequence, append the result of applying UTF-8 encoding to character to bytearray".


The draft says "Implementation of "100 (Continue)" support is OPTIONAL for clients and REQUIRED for proxies." What does that mean? Is the client allowed to fall over if it receives 100? (This is not a theoretical concern, many HTTP libraries stop listening for header blocks after the first one.)
I've posted to reduce ambiguity here. RFC 9110 I think that makes it pretty clear that clients are supposed to tolerate unexpected 100s.


This treats empty dictionary strings as the same as there being no dictionary and reqires all ID's have a non-zero length for the Dictionary-ID header to be sent.
NAME could you PTAL and see if this handles the case you were concerned about?
Dictionary-ID was introduced to the Compression Dictionary Transport specification by . However, it is unclear whether empty string dictionary IDs are allowed. I don't think there is any use case for an empty string dictionary ID. So how about explicitly disallowing empty string dictionary IDs in the spec? NAME
I'll make a zero-length string the default and have that mean "no ID". The dictionary ID will only be sent if it has a length > 0.
Thanks. lgtm.


This changes the field to be a list of destination strings to allow for matching multiple destinations (i.e. document and frame). This allows for matching an empty destination explicitly using an empty string. e.g.
NAME could you PTAL and see if this looks OK for the destination matching?
The spec of compression dictionary transport says: But an empty "" is used for some features such as , , , , , Cache API, Download, prefetch and prerender. (See this ). So currently there is no way to set up a dictionary for such destinations. For example, I want to use a dictionary for fetch() API, but I don't want to use the dictionary for HTML documents. NAME How about using a string "empty" for such empty destination? We are already using "empty" for header of empty destination request. URL Also if we don't need to use an empty string "" for , I think should be a instead of a . We don't need to use double quotes.
Having special values feels a bit strange, particularly since it's only defined in the internal processing part of the w3c spec. If we use a list of strings we can use for an explicit empty destination and use an empty list for matching everything (and as the default). Does that seem reasonable?
OK, sounds reasonable. Thanks.
option was introduced in the Compression Dictionary Transport spec by . But if my understanding is correct, currently it is not allowed to setting multiple types of destinations. So if I want to use a dictionary for HTML files including top frame, and , I need to download the dictionary three times. NAME How about allowing setting multiple types of destinations? Example:
Sounds good (though document/frame is probably the only practical use case). I'll switch it to an inner-list of tokens.
lgtm Sorry for the late reply.


Switched from using the ABNF field types to the human names (i.e. sf-dictionary to Dictionary) for all of the Structured Field header values. Switched dictionary "type" to a Token (from String).
NAME could you PTAL at the change in referencing the header field types? Specifically, I wasn't sure if I should capitalize the types or not (I did since that's how the structured fields RFC referred to them).
Yep, this looks good. It's common to add something to you Notational Conventions section (or whatever) that says roughly "This specification uses the following terms from [ref to SF]: String, Byte Sequence, Token, ..."


Changes: s/tcpport/targetport/ (aligning with connect-udp) Remove support for multiple IPs and
NAME The discussion at IETF 119 leaned toward removing the multiple-IP mode entirely. Are you suggesting that we should keep it?
This PR contains two changes; I'm not making a suggestion with regard to multiple-IP mode, but I think aligning the variable names in the template makes sense. Did both get dropped, or is that being discarded because it's in the same PR?
OK, I've updated this PR to also drop the multiple-IP support entirely.
The target_host variable is only described vaguely. It should be defined more rigorously. For example, it doesn't specify whether IPv6 scope IDs are allowed or not. I'd recommend reusing the text from RFC 9298.
OK, I've opened to make the definition more rigorous.
Fixed by
The draft currently defines the "targethost" and "tcpport" variables. We should replace "tcpport" with "targetport" for consistency with connect-udp. This consistency isn't just cosmetic, it simplifies reusing the same URI template for both protocols.
The distinction is deliberate, precisely because it facilitates using a single URI template for both protocols. As discussed in , the client can use a single template for various purposes by inspecting the variables that it contains: "targetport" -> connect-udp "tcpport" -> connect-tcp "ipproto" -> connect-ip "dns" -> DoH Using the same name ("targetport") for TCP and UDP would add complexity for client implementors. Instead of being able to determine a priori_ whether a proxy supports TCP and UDP based on the template, the client would have to probe (and periodically re-probe) for this information and maintain dynamic state about which protocols are supported.
What you're saying is that a client can infer support for certain protocols based on the variables present in the URI template. That's not correct, or more precisely it wasn't specified for connect-udp, and it definitely doesn't work for connect-ip (in connect-ip the variables are optional).
OK, I see that it doesn't work for connect-ip, so connect-ip can only be supported by probing or with an out-of-band signal (or with an ecosystem restriction that connect-ip servers must support "target" and "ipproto"). It seems fine for connect-udp and connect-tcp because the variables are mandatory, so it would provide value to UDP+TCP clients (which seem likely to be common).
In that case we should state somewhere that passing around an URI template with these variables implies support for the corresponding mechanism. I'm not sure we'd get consensus on that though.
Thanks for the updates based on the discussion at 119


This PR adds descriptions to all message exchange examples in the draft. It should make their purpose clearer and also help distinguish the two separate examples in section 4, as pointed out in .
Thank you all!
نزت
I think there's two different creation examples being shown in URL but the presentation of them isn't great and could be a source of confusion. If so, lets break them apart by having a short sentence before each example expliaining it.
Yes, these are two separate examples, which are confusing. Most of the example are in need of a proper description. I can open a PR to address this. Thank you!


This PR adds explanations helping readers to understand when empty requests for upload creation or appending might be useful and why server should support them. Closes URL
The field definitions allow for a client to send 0-length content. Sometimes this is useful, other times it it might not be. For example, A client could perform upload creation by making a request content-length: 0 and with upload-complete:?0. This would allow it to test creation wihout having to commit to an upload in the same message. A client could attempt to append to append zero bytes. Is this useful or should we prohibit it? This is a cheap way to keep an upload "active". However, that might be a vector for abuse by allowing a client to cause server-side state commintment. Depending on what we decide, we might need some text in the security considerations.
Just to add to the two bullet points: This can also be used for clients/servers that do not support 1xx response by sending a 0 byte upload-complete: ?0 request to get the unique resource URL. This is for instance how does it. For cases where multiple upload-complete: ?0 requests are made, a final 0 byte upload-complete: ?1 request can be used to notify the server that the upload is complete. This can be useful in cases where the client wants to provide all the data before finalizing the upload to reduce the risk of losing the final response. I don't really see a use case for sending multiple 0 byte upload-complete: ?0 but there might be some?


A server might allow GET requests to be sent to an upload resource and serve content, in which case a 200 status code is more appropriate for responses to HEAD requests. This PR lifts the requirement that a successful response must be a 200.
The current draft requires the server to respond with a 204 No Content status code for HEAD requests: [...] If the server considers the upload resource to be active, it MUST respond with a 204 (No Content) status code. This restriction is too strict and does not allow the server to respond with a 200 OK. Since the HEAD requests for resumable uploads do not include any specific additional header field, the server cannot tell apart whether a HEAD request is used for resumable uploads or used for starting a download, for example. For example, a server might allow to download the uploaded file through a GET request to the upload URL. HEAD requests to this URL should then return the same status code (e.g. 200) and header fields as the GET request would do. But the draft requires a server to always respond with 204, leading to a conflict. Can we also allow 200 for HEAD responses?
I think this would be a good change to the spec. Maybe use similar language as with PATCH, where we allow the entire 2xx range but have a recommended status code? I don't think this would be an issue for any client. Is this a typo? Should be "... 200 for HEAD..."? >Can we also allow 204 for HEAD responses?
That's correct. Thank you for pointing out


allows another status code to be returned for offset retrieval requests. This might break interoperability between existing implementation, so this PR bumps the interop version to make this change more visible.


This PR removes the requirement of checking the request's redirect chain during the computation of same-site-ness. This is being done because RFC6265bis is blocked by this work but we have yet to find a way to implement it in a web compatible way. In the interest of moving RFC6265bis forward the requirement is being removed.
The work to re-add the requirement back into RFC6265tris is being track by issue
Yeah, we did implement this - but we had to back it out (and re-spin stable, IIRC?) because it broke too many sites. :(
Correct, Chrome had to disable the change. Firefox had a similar experience.
We noticed a difference between Firefox & Chrome’s behavior for same-site cookies. Specifically, if a web page is requesting a resource whose final redirect target is same-site with the web page, . Firefox follows what was originally decided in URL and specified in recent i.e., Firefox looks at the whole redirect chain. For context: Firefox shipped SameSite=Lax by default this January and had to backpedal due to this (and some other webcompat issues). Firefox does not ship SameSite=Lax as the default setting for new cookies as of now. While we’re still working on our own metrics, we’ve noticed that Chrome is seeing about 1% of page loads potentially affected, which seems prohibitively high. At the same time, we would prefer not to weaken our implementation of explicit SameSite cookies, given that the redirect check was specifically added to prevent CSRF abuse scenarios. With these things in mind, we’ve been exploring the idea that we might treat cookies differently depending on whether the cookie was set to lax by default or is an explicit samesite cookie. NAME NAME it would be nice to hear some thoughts about this idea and also NAME how other browsers may have implemented . CC: NAME NAME NAME NAME NAME NAME
Yeah, I agree this is unfortunate. We did , but it broke a few sites and was backed out (or rather, put behind an off-by-default feature flag). NAME has been collecting better metrics so we can hopefully make progress on this. Can you give some more detail what you're thinking here? (also cc NAME

With these things in mind, we’ve been exploring the idea that we might treat cookies differently depending on whether the cookie was set to lax by default or is an explicit samesite cookie. Firefox is already enforcing that all redirects are same-site for cookies that have an explicit or strict attribute. We would like to keep that restriction of course. From my understanding the idea would be that only cookies that don't define their own SameSite attribute and are then "lax-by-default" we would ignore the redirects.
As a side note, Chrome does send with these redirected sub-resource requests, so internally the networking code does seem to know the correct same-site value of the request. Obviously we'd like to do the more secure behavior all the time, but we are committed to doing the specified secure behavior for explicit SameSite cookies. Firefox has enforced this for explicit SameSite cookies for maybe around a year and has not suffered a noticeable amount of web incompatibility. We are in the (slow!) process of gathering telemetry to measure the impact of this on laxByDefault vs explictly SameSite cookies. Our hope is that we can convince Chrome folks to honor the spec for explicit SameSite cookies so that users and sites will be secure as promised by the spec, and that Firefox will be less likely to run into Web Compatibility problems in the future because of this. And that if we must weaken the behavior of laxByDefault (as it appears we must based on site breakage) that we explicitly change the spec to say "lax-ish by default" cookies follow Chrome's current behavior of ignoring redirects and comparing only the originator and the final request URL.
Hello and sorry for my late response. Our (Chrome's) metrics continue to show around 1% of page loads containing 1 or more cookies that would be blocked by this enforcement. While I'm unable to share numbers (yet, I'll add them to URL once they're approved) I can say that the vast majority of values triggering this are unspecified (lax by default). I wonder then if many of these cookies are likely those being accessed after a POST request from some login or payment flow redirect chain. I don't have any data to back this hunch up but URL hints this may be the case. If it is then perhaps we can tie the redirect chain enforcement to the 2 min Lax+POST mitigation. I.e.: Allow unspecified cookies to be accessible after cross-site redirects for up to 2 minutes. While I worry this will further ossify that "temporary" mitigation it feels better than a blanket except for unspecified cookies. Given our metrics I don't think your proposal is unreasonable, but the more we can tighten the restriction the better. Any thoughts? Does Firefox have any data that could support or refute my idea? (Tangentially: I'd still like to see the 2 min exception removed, but I get a headache even thinking about it.)
tagging NAME NAME for visibility to the above comment. I think tying this to the 2 minute exception is fairly reasonable - 1% of page loads is sadly much too high to ship what the spec describes in any reasonable time frame. Perhaps some pages would still break, but it's worth a shot. And I agree we should change the spec to match (whatever becomes) reality here.
What about the idea of differentiating between explicit and -defaulted ? Not ideal, but then at least there won't be a security regression for those who opt into . Then for -defaulted we could do something along the lines you propose, in name of webcompat.
Right, we already differentiate between explicitly Lax () and unspecified () when it comes to the 2 min Lax+POST mitigation. Only the unspecified cookies qualify for that mitigation, explicitly Lax do not. (Perhaps it would have been better to name it Unspecified+POST, oh well) I'm suggesting to continue that pattern for the redirect chain exception, so I think we're in agreement.
NAME Yes, that's aligned with our thinking right now. Do you have numbers that can help us gain more clarity for these exact cases (a redirect where the final URL is same-site, but the overall chain is to be considered cross-site, also called "boomerang redirect") and split up into explicit-lax and default-lax? We're still in the process of collecting those within Firefox, but are obviously interested in your insights so far.
NAME I think URL has what you want. Those numbers are from roughly a week ago. tl;dr Between 80-90% of cookies are unspecified/default lax. Cookies that are being newly written are on the higher end of the range, cookies being read are lower.
Hi! John from Apple WebKit here. If we standardize different behavior for default vs explicit SameSite=lax … We increase the complexity of SameSite cookies significantly. Is that desirable? Will developers be able to know the difference without this effectively being displayed as an additional cookie attribute in developer tools? Are we arriving at the desired end state or piling on more legacy and technical debt? It looks to me like this should instead be a fourth SameSite value, beside none, lax, and strict. Then we’ll have to decide if the new, fourth value is the default or the old explicit lax. I generally don’t like fixing legacy problems by increased spec complexity since it doesn’t help the web/Internet progress. I will bring this up with the CFNetwork folks at Apple to see what they have to say.
Hi John, You bring up some good points. I'm curious if the CFNetwork folks had anything to add.
I think what's under discussion is making the fourth value explicit in the specification. It apparently is already a thing in some implementations. I'm not sure we have a good reason to expose it in syntax though. But yeah, a clear name in the specification would help with tooling and such.
+1
"Default" is an explicit value for the in the algorithms and storage model defined in Section 5. It's not a part of the syntax. Unfortunately it looks like the algorithm only sets the if there's an explicit SameSite attribute in step 17 ("Default" is used for an otherwise unknown value). Presumably the should be initialized to "Default" somewhere, but that's an assumption and not explicit. The value of same-site-flag is technically undefined if there isn't a SameSite attribute and that's clearly not the intent.
Hi, Zhenchao Li from Apple working on CFNetwork. This change sounds quite complex to me. Although, if it means minimizing website breakage and tightening cookie restriction, I'm in favor of explicitly adding the behavior to the RFC. It does sound like we need to move from "SameSite=lax by default" to "SameSite=[fourth value] by default". Also, "default" still needs to be a separate state persisted in "Storage Model" so as not to be conflated any time. (I believe this is intended by latest draft, but as NAME pointed this might need some changes to the Storage Model section) NAME Could you help clarify more about "2 min Lax+POST mitigation"? I see it mentioned above but it would be great to discuss how the mitigation should behave exactly.
URL is the spec language around the behavior. As a quick refresher: the goal is to allow unsafe http methods (commonly POST) to attach cookies that do not specify a attribute but are being treat as "lax by default". This behavior is intended for newly created cookies and so has a lifetime limit of up to 2 minutes. Let me know if you have any more specific questions.
We (NAME NAME NAME and I) discussed this at TPAC and came to the following suggestion: We think that browsers have to apply a compatibility measure here, and it could be including cookies on cross-site redirects if they’re younger than, for example, 2 minutes. We’ll collect metrics to get a better understanding of what that number should be. We will update "Lax-Allowing-Unsafe" in the spec to document that behavior, which would make Chromium compliant to that and Firefox could (AFAIU) ship a compliant version as well. We’ll continue to recommend "Lax-Allowing-Unsafe" as the default behavior in the spec. We’ll note that this is a compatibility mechanism that browsers may support or not support at their own discretion. It's not a behavior sites should opt into.
I believe the "same-site redirect chain consideration" change should be removed from RFC6265bis, with the intention of adding it back to RFC6265Tris. RFC6265bis contains a number of improvements over RFC6265 that are being stalled due to UAs being unable to implement "same-site redirect chain consideration" in a web compatible way. My efforts examining metrics collected by Chrome haven't yet offered much insight into how this issue could be solved and I have no idea how much longer it could take. Because no UAs (to my knowledge) have shipped "same-site redirect chain consideration" I don't believe reverting the language will have any practical effect and shouldn't weaken any active protections. But publishing RFC6265bis as a new spec will have a number of benefits, including security/privacy benefits for the web's users.
This seems like a reasonable, pragmatic step to me. The current limbo state isn't helping anyone :) cc NAME
Yes, +1 to documenting reality and no longer blocking the IETF process for 6265bis on this particular item (which I do agree we should treat as a security bug to be eventually solved).
Not an editor, but LGTM (given that it reflects reality).LGTM, given the context.


Technologies such as HTTP/2 will split the header into multiple fields. This PR modifies the spec to no longer expressly forbid this and instructs servers to be aware and capable of handling multiple incoming headers.
In the original 6265 as well as in the latest 6265bis draft, this paragraph exists: This is directly contradicted by RFC 7540 and then : I personally suspect that trying to split the header up into multiples is a recipe for disaster and interop problems even if done only over HTTP/2, but maybe I'm just a pessimist. Would it be sensible to maybe address this contradiction (better) in the bis document?
HTTP/3, repeats the HTTP/2 statement as well: Both these HTTP RFCs also clarify that: But still, that is for the receiving end and does not really restrict the sender when using HTTP/2 or HTTP/3.
I agree that the inconsistency is bad. That said, why do you believe that using multiple field instances can cause an interop problem?
I'm fairly sure Chromium does this, so if there were interop problems, we'd probably have noticed by now. :-) The phrasing is a little odd and probably could be improved. One resolution is to say this is an HPACK-specific encoding thing and not actually sending multiple fields. The full text from 9113 says: I.e. this field splitting starts and ends its life in HTTP/2.
A concern: Some servers have a maximum request header field size limit of around 8K. curl makes sure to never send a header longer than that as it risks getting a 400 back and basically blocking that particular request without it being very obvious to the user. This is a limit RFC6265(bis) does not mention but probably should because it is one of the more important ones. By splitting up the header into multiple ones, the total size of the sent cookies should probably still remain smaller than 8K since there is a risk that there is a h2/3=>h1 conversion done that will pass those cookies on in a single header and thus be affected by this limit. It would if HTTP/3 didn't have the same wording so it also applies there.
Because one RFC says we MUST NOT send them and another says we MAY. I bet there is a server side or two out there that then will not appreciate getting multiple such header fields, perhaps because the h2/h3 special case is not taken care of. But this is just be being worried, I don't know for sure this is the case.
Agreed that such a limit should apply to the total here. I think that would follow from treating this as a funny HPACK encoding of a single field, rather than semantically sending multiple fields. That's not precisely how it's described today, but it results in the exact same behavior on the wire. I.e. we can still get the compression benefits of compressing individual cookies separately, and resolve the slightly awkward phrasing.
Safari started splitting Cookies in h2 and h3 a few years ago, it actually improved website compatibility where the h2 load balancer had a per-field size limit but the ultimate h1 origin did not.
Understood and agreed. So yes, that "MUST NOT" doesn't make any sense, and it's inconsistent with the base HTTP spec. Would be interesting to find out how that got there.
I can't recall the discussions about it on the http-state list back in the day (man, it's been over a decade already!) and I could not find any relevant threads when I casually searched the archives.
Changed from SHOULD NOT to MUST NOT between draft 09 and 10: https://author-URL
I think the wording is intended to be internal to HPACK/QPACK -- that is, there can only ever be one Cookie field as far as HTTP is concerned, but Cookie is special-cased to be split when encoded. To deal with the special case, any occurrence of multiple values gets coalesced. I don't see this as a contradiction.
Are there any remaining concerns before I close this issue? I agree with Mike's interpretation that this isn't a contradiction. The spec mandates that UAs generate a single header field which the server expects on the other end. I don't believe that another layer transparently muxing/demuxing this field is an issue.
I agree that Mike's interpretation is the sensible one to draw from this. I disagree that this is what the specs actually say in the written words.
I can see the potential for confusion here. This seems to stem from 6265bis's tendency to re-implement other specs requirements, but in a somewhat inferior way. The cookie spec probably shouldn't care how the HTTP request is specifically formed so long as the invariant of a single cookie-string from UA -> Server is maintained by the time the server's cookie processing code receives it. URL would probably help here. In the meantime, I'd prefer if we didn't special case actions such as 9113's cookie header field splitting, so maybe some rephrasing could avoid the issue? Here's a rough draft of what I mean. My intention here is to make it clear that the UA (more specifically the cookie code) should create a single cookie-string to attach as the header's value. This would lead to a single header in the default case, but should still allow for header splitting.
9113 and 9114 have to special-case Cookies for one simple reason: they're a semicolon-delimited list rather than a comma-delimited list. 9110 ; the text in question applies the same logic to Cookies despite the different delimiter and points out that for compression reasons it's particularly useful to split Cookie fields. (It might have been worth noting that other list fields might see similar gains from being split and recombined across the compression boundary, too.) I think NAME proposed text on generation is fine -- there should always be exactly one Cookie header generated from UA->server, though a lower layer might mux it before compression. After decompression, demuxing is mandatory when being passed into a context that doesn't know to expect being split. However, there's that exception in both RFCs: "URL being passed into a non-HTTP/2 context, such as an HTTP/1.1 connection, or a generic HTTP server application." This text clearly supposes that anything version-specific which comes after RFC 7540 will know that Cookies may have been split; perhaps this document should prescribe tolerance of receiving that on the server side. It's then an implementation concern whether that tolerance takes the form of recombining them before processing.
Which document specifically?
Sorry, let me clarify antecedents: >This text [found in RFCs 7540, 9113, and 9114] clearly supposes that anything version-specific which comes after RFC 7540 will know that Cookies may have been split; perhaps [6265bis] should prescribe tolerance of receiving that on the server side. That is, perhaps the Cookie spec should require that servers be able to handle Cookie fragments having been split into multiple Cookie headers, since this document has the opportunity to be aware of HTTP/2+ behavior.
Thanks for clarifying, that makes sense.
LGTM % tiny nits.