Adds some additional notes and warnings to developers on how to more carefully use SameSite cookies.
We are reporting on the behavior of cookies on cross-site windows opened through that goes against our assumptions (implied by the current specification) on when and how cookies should be isolated. Breaking these assumptions can lead to security issues, allowing the bypass of cookies as a defense mechanism against CSRF and XS-Leak attacks. The following table shows when cookies are attached to requests associated with s and pages opened through (where :x: stands for not attached/not accessible and :whitecheckmark: for attached/accessible): The behavior of cookies regarding page reloads for pages opened through from a cross-site position is evident from the : Currently, the specification mandates blocking cookies on user-initiated page reloads but not on reloads performed through JavaScript. We think that cookies should also be blocked in this scenario, as this interaction between JavaScript navigations and cookies allows attackers to bypass the restrictions imposed by (see 2.1). Additionally, we verified that cookies, even when not sent in the top-level request that loads a page, are still attached to requests that load subresources for that page. This behavior is not explicitly defined in the specification. However, its counterpart (setting a cookie) is defined as follows: Although these requests are not cross-site, they are performed implicitly by a page navigated from a cross-site context, effectively enabling an attacker to perform authenticated cross-site requests (see 2.1), or cookie tossing attacks. Following this argument, when considering that user-initiated page reloads are deemed cross-site and therefore do not attach cookies, it seems reasonable to assume both JavaScript navigations and subresource loads should also not attach cookies, as they are also transparent to the user. It is possible to abuse the previously described behavior to bypass CSRF protections An attacker can forge same-site requests starting from a cross-site position by abusing gadgets that result in secondary same-site requests. An example of one of these gadgets is a JavaScript-based redirect that performs a redirection using attacker-controlled data. Here is a possible attack flow: Victim visits URL, setting the cookie cookie for domain . Victim then visits URL, which executes The request to URL is cross-site, so it does not attach cookies URL performs a navigation to the value of the URL parameter through This triggers a request to URL, which attaches cookies, allowing an attacker to make authenticated requests from a cross-site position A similar scenario was discussed in this . Besides JavaScript redirections, other requests, such as subresource loads, could also be exploited to subvert the security assumptions granted by cookies. cookies effectively defend against certain classes of XS-Leaks attacks. However, -based attacks are still possible since cookies are attached to top-level cross-site requests. Since cookies are not attached to top-level cross-site requests, they have been deemed an effective mitigation against -based XS-Leaks. As discussed in 2.1, gadgets that allow attackers to forge same-site requests from a cross-site position can be leveraged for XS-Leaks. An example would be a page that performs a stateful subresource load, which modifies the page DOM. If an attacker opens this page through , cookies will not be attached, but following subresource loads in the new page will attach these cookies, leading to state-dependent changes to the DOM, which could be measured by the attacker (through frame counting for example). Best regards
Thanks for your well researched issue! I’ll try and address some of the points you brought up. First and foremost, SameSite is meant as only one layer in a defense in depth strategy, its design and implementation tries to balance our security and privacy goals with the realities of the web as it exists today. This ultimately means that it must be used with smart site design that leverages the other security tools and features available. differs from the original site (let’s call it a.example) navigating itself by allowing a.example to continue to hold and manipulate a reference to the new window. But, every navigation a.example makes on that reference will have an initiator of a.example (On Chrome at least, I haven’t checked other browsers, but I believe it’s true for all). This means that, as far as SameSite is concerned, an a.example window navigating itself directly or opening a new window and navigating that instead are all the same type of top-level navigation. This appears to hold true for all the examples and notes in your post. This was a purposeful decision. We wanted to allow the site to be able to operate specific “landing pages” in order to allow third-party entities (maybe a sign in flow or payment processor) to navigate the user back and then the landing page could refresh to allow access to cookies in order to complete the process. Since the reload must be explicitly initiated by the page it would require the site developer to “opt-in” to the behavior and hopefully design the landing page with that behavior in mind. Conversely, we disallowed user initiated reloads because that doesn’t show an intention on the site’s behalf that it wants or needs cookies following that cross-site navigation. Users have been taught over the years to try reloading if a page looks broken, something that could occur if cookies aren’t present. Continuing to block cookies helps prevent users from inadvertently weakening their own security. That’s expected and intended. Samesite operates on a per request basis (or on a per JS call basis) so all requests, sub-resource or otherwise, are subject to the same rules. See . The spec assumes that since the page loading the same-site resources is in control of what it loads, it shouldn’t load anything that makes harmful changes. You’re correct that if a site exposes an API like you’ve given in 2.1 then it’s possible to give an attacker control + access to SameSite cookies. But to me it appears that the gadget is the footgun and I’m not sure how/if SameSite should solve this. Allowing any cross-site entity to make requests as yourself is a dangerous tool! I mentioned above that pages are in control of what they load and as long as they only load safe resources that they’re expecting they should be fine, but if you have any examples to the contrary I’m very interested in hearing about them. Going back to how s are basically the same as any other top-level navigation, this is the intended behavior. Lax cookies shouldn’t be used for anything that can allow changes to anything important. Agreed, it’s definitely possible to leak state this way. I’m not familiar with the reasoning with why the single-origin policy allows references to read , but I’m sure the spec writers had a good reason. Hope that helped clear things up, let me know if you have any other questions or concerns.
Hi NAME thanks (as usual) for the kind and thorough response! I did a bit of additional investigation and found some discussions that we missed before opening the issue: URL and URL As far as I understood, the current behavior concerning page reloads has been defined only recently (2021). Before that, different browser vendors had . Even nowadays, browsers do not seem to fully agree on the intended behavior, or at least Firefox is inconsistent with Chrome's behavior since it does not attach SameSite=strict cookies after a call to . Some confusion on this matter is justifiable. :) I now see the rationale behind the design that enables specific pages to opt-in to SameSite=strict cookies upon a self-initiated reload. I always thought SameSite=lax cookies ensured compatibility with SSOs and payment flows, as lax cookies are attached to top-level navigations. But this "strict + reload hack" (so to say) puts the webpage in control instead of attaching cookies to all cross-site top-level requests. The downside of this behavior, as discussed, is that JS-based open redirectors become a gadget for bypassing SameSite=strict cookies. I'm not sure if the benefits outweigh the risks, but probably, the spec would benefit from a more explicit discussion of this trade-off. Concerning subresource loading, I would argue that in general this is not a valid assumption. In principle, here we have an untrusted context (the cross-site page opened via ) that automatically loads authenticated resources. This is somehow the dual of SRI, CSP, mixed content, and other security mechanisms that prevent a trusted context from loading untrusted resources that an attacker could abuse. We have not measured the prevalence of security issues caused by this behavior in the wild (i.e., xs-leaks introduced by an authenticated subresource being loaded), but I would expect a non-negligible number of affected instances. The current behavior surprised me, and it is possible that developers who care about protecting from xs-leaks are also unaware of the potential risks. For instance, the states that: citing only other reasons (e.g., the 2 minutes exception rule) as major limitations of SameSite cookies. To conclude, we just wanted to discuss this issue after encountering a behavior that diverges from our mental model of SameSite cookies and that we could not entirely justify it after reading the spec. As for JS-initiated reloads, if the behavior is here to stay, I think the spec could mention these limitations more explicitly to prevent developers from building their applications based on wrong assumptions. Thanks again for the discussion! Marco
Hi NAME As you found in those discussions, reload behavior wasn’t well aligned previously and I’m not sure how purposeful each UA’s behavior was at the time. I know Firefox has a number of SameSite related behaviors behind default disabled configs, this could be as well. I don’t have easy access to Firefox at the moment but I believe the nightly builds have these flags enabled. What happens if you try testing on a nightly build? SameSite=Lax is useful for flows that use top-level navigation, but if a flow is happening in an iframe then SameSite=Lax cookies would be blocked. The reload technique enables these iframe based flows to function. The issue here is that a page may still load sub-resources that require SameSite=Strict auth cookies even if the initial request didn’t include those auth cookies? Plainly, any site that allows unauthenticated access to a page which loads authenticated sub-resources is incorrectly using SameSite. Sites should return a 401 error, or some other denial, if the initial request does not include the appropriate cookies. Sadly though, I agree with you that there are probably sites that do not. Just to confirm, the behavior you’re talking about are: URL navigations are not as restrictive as iframe navigations. Non-user controlled reloads allow cookies Sub-resource requests send SameSite cookies Redirect gadgets allow access to SameSite cookies Did I miss any? What do you mean by “these limitations”? Are you referring to the redirect gadget?
Any remaining comments/concerns before I close?
Hi NAME Sorry, I missed your earlier reply in October, and I'm just now seeing it. Thanks for the ping. I tried Firefox 123.0.1 (stable) and Firefox 125.0a1 (nightly), and they behave the same way. SameSite cookies are not attached to requests initiated via , but they are attached to requests caused by . Chrome sends the cookie in both cases. Indeed. While open redirectors and script gadgets are known to carry their problems, I am concerned about issues enabled by authenticated subresource requests, as they are less known to web developers. I don't think so. All of the above, actually. I understand that it's challenging to change the behavior at this point. Still, I think it would be beneficial to explicitly document both redirection gadgets and authenticated subresource requests under Sec. 8.8 of the spec. This would help web developers to understand the risks better and adopt mitigations. Cheers, Marco
Still LGTM.


Rephrases the requirements for the attribute to be slightly broader in order to support . Currently the requirements specifically ask for a secure protocol, with this change they'll ask for a secure connection (as defined by the user agent). This allows UAs to support cookie access on http://localhost
No WPT that I can see. According to , Firefox fully supports localhost, Chrome supports but not cookie prefixes, and Safari doesn't have support but does have an open bug.
Got it, thanks. If you're taking care of Chromium, and WebKit has an open bug, landing this seems reasonable. Thanks!
We are reporting inconsistent browser behavior regarding cookies and and prefixed cookies on . delegates the decision of what is a "secure protocol" to the user agent when describing how to handle cookies with the attribute. The affects both cookies with the attribute and cookies with name prefixes ( and ). rfc6265bis-12 states the following (): According to our tests: Firefox allows cookies to be set/sent on , as well as and prefixed cookies. Chrome also allows cookies to be set/sent with the attribute. However, and cookies are not allowed. Safari does not allow cookies to be set/sent, and consequently, and cookies are also discarded since they require that the cookie's be set to true via the cookie attribute. The table below summarises browser behavior when setting cookies on , localhost subdomains, and . We noticed these inconsistencies after interacting with developers of Web frameworks while discussing cookie integrity issues. Allowing cookies with the attribute and cookie name prefixes to be used on , would enable adopters of Web frameworks to test their applications locally, removing the burden of setting up HTTPS locally to use these security features. As shown in the table above, Firefox fully supports this behavior, while Chrome's support is still partial. Following , Firefox moved from a "secure protocol" check when handling cookies, to using the definition of specification, where is considered "Potentially Trustworthy". The states that: >An environment is a secure context if the following algorithm returns true: Chrome made the same decision following , and support for and prefixed cookies on has been deemed "in scope" (). Safari is aware of this inconsistency, but there is still no decision regarding changing their behavior (, , ). As the cookie specification does not clearly define the behavior on localhost, we could consider reusing the notion of from the Secure Context specification, instead of suggesting a user-agent-dependent "secure protocol" check on the scheme of the request-URI, in point 13 of . Best regards
I think it's reasonable to modify the phrasing to include potentially trustworthy origins.
This is a good change, thanks. I don't think we have WPT in place for this (and I'm not entirely sure that we can?), but have you looked into the support for this change across browser engines? I'm generally in favor of landing it as long as it's reflecting (near) reality.


There were a few cases of the section linking to itself, which I find distracting, so lets remove them.
Good catch, thank you!


Please correct me if I'm wrong, but IIUC, the crux of this PR is that to allow any client that can present a valid pair of Authorization and Signature-Auth-Context header fields to obtain a protected resource. Assuming that if we would be allowing such behavior, I wonder why we need to require clients to export keys from TLS session, rather than just coming up with some arbitrary authentication context and present it to the server? Or if it is the case that there are situations in which we have to require clients to use keys exported from TLS session, doesn't it mean that there are cases where servers must ignore Signature-Authentication-Context header field or drop it?
NAME no this is only for trusted intermediaries. If a server receives a Signature-Auth-Context header from an untrusted source then it drops it. That's somewhat mentioned on line 470, but I'll add normative text.


This change makes it explicit that the content encoding is (same as ) and is (same as ). See the discussion in for why larger ZStandard values were not selected. This carries some nuance for what encoding to use when compressing for maximum benefit as the resource size crosses beyond 8 MB but that is more of an editorial discussion about the individual compression algorithms and probably better laid out in blog posts or explainers than in the spec.
NAME could you PTAL and make sure the language matches what you're working on for the update?
Thanks. I switched the content-encodings definitions and IANA registrations to a list with references back to the algorithm RFCs. For the IANA registrations I removed the window size from the text and linked it to this document (where the window sizes are defined) instead of to the compression algorithms directly. Let me know if a more verbose description of the need to specify the window sizes would be useful or if just being explicit about what they are is enough.
Context: URL aims to clarify the window size limit for the Content Encoding token as it wasn't clear in RFC8878. Since the Compression Dictionary Transport draft hasn't been published yet, we should also add some window size considerations to it for and to maintain interoperability. I think in Chrome we ended up using . NAME
NAME could you comment on how this impacts Brotli dictionary decode? Is the dictionary independent from the compression window (so we can adopt the same defaults as or do we need to specify something relative to the size of the dictionary? NAME NAME for ZStandard, to allow for delta-encoding large resources, is reasonable or should it be ? We have use cases for delta-encoding wasm applications where the dictionary can easily be upwards of 50 MB.
For Zstandard, the states that the entirety of the dictionary is accessible as long as the input is <= Window Size. Therefore, to give an extreme case, it's possible to have a 50 MB dictionary, and it's entirely accessible even if the Window Size is only 1 KB, but only as long as the data to de/compress is <= 1 KB. As soon as we reach 1 KB + 1, reaching the dictionary is no longer valid. For delta compression, this has consequences. Presuming the reference is ~50 MB, and the updated content is also ~50 MB, one would need a window size of ~50 MB to ensure access into the dictionary during the entire reconstruction process. But if the updated content is just a bit bigger, say ~51 MB, a ~50 MB window size would allow delta operation only during the first ~50 MB, the last ~1 MB would have to be compressed without dictionary. If follows that the size of the dictionary is irrelevant for the Window size. What matters is for how long one wants to access the dictionary while rebuilding the content. And as a general approximation, one wants this dictionary during the entire reconstruction process, so the window size is effectively the size of the content to rebuild. Zstandard has a "windowless" mode which states exactly that : the size of the "window" is the size of the content. For this to work, one has to set the size of the content at the beginning of the compression process . Setting the content size is either an automatic operation when data is provided entirely upfront (), or, when data is provided in small increments (), it must be stated explicitly at the beginning (). will automatically trigger the "windowless" mode when the declared content size is smaller than the set Window size (effectively reducing the memory requirement for the decoder side). I'm not sure if there is another way to trigger it. Note that the decoder will be informed of the Window Size requirement by reading the frame header, and may decide to not proceed if this value is deemed too large. For example, has a default 128 MB limit, and will refuse to decompress payload requiring more that that, unless explicitly authorized to employ more memory (). Just for reference, offers a "large delta compression" mode with the command . It has made us somewhat familiar with some of these issues. In situations where the amount of data to compress and compare is very large, we now prefer to both the dictionary content and the destination file. It has proven useful to reduce memory pressure.
Thanks. That makes things a bit more complicated for the case where we want to require a minimum window size that the server can trust that all clients advertising content-encoding will be able to decode. For the dictionary case it might be cleanest to use the same 128MB limit for window sizes as the default zstd limit (knowing that the client won't allocate that much unless it actually gets a stream that uses it).
Hello. Shared dictionary is placed in the address space between backward reference window and static dictionary. In other words it does not interfere with normal compression window, but using it makes "static dictionary" references more expensive. We decided to order it that way, because shared dictionary works almost the same way as extra window. Sorry for the late response. Best regards, Eugene.
Extra note. As you can see brotli allows backward distances behind window. IIRC, regular (as opposed to large-window) brotli allows distances up to ~56MiB, i.e. larger shared dictionaries would not be useful (== referenced)
Thanks. For the dictionaries to be usable for a broad set of apps (including large wasm bundles), I'm considering spec'ing: Brotli: Regular 16MB window, allowing for delta-compression of resources up to ~50MB. ZStandard: 128MB window, allowing for delta compression of resources up to around that size, depending on where the duplicated bytes are in the updated resource. The main difference with regular content-encoding is on the ZStandard side where the window is 8MB. For platforms that are resource constrained and are not willing to support a 128MB window they would be limited to Brotli (and then decide how large of a dictionary they are willing to keep and use). Another option on the ZStandard side would be to allow for a maximum window that is a multiple of the dictionary size (i.e. 8MB or 2 * dictionary size whichever is larger with a 128MB max). That way the client can have an upper limit that it knows in advance before advertising as being available. That adds complexity to the encoding side though. We have already seen cases where people want to delta-compress resources > 50MB (wasm apps) where they had to use ZStandard instead of Brotli because of the Brotli limits so I don't think a small limit like 8MB for ZStandard would work with this use case.
That massive resources exist is not sufficient justification to build a system that will not be available to some users. I would prefer a far smaller default. If only to encourage the use of smaller resources overall. Defining a separate codepoint for massive windows is possible, say . The scalable option also seems reasonable. Clients can then choose to offer dictionaries when they exceed available resources.
I'm ok using the same 8MB window ( I believe) that we use for for and note that the effectiveness of delta compression for responses > 8MB using may be limited. The scalable option concerns me that people are likely to get it wrong and it won't be obvious and the decompression will start to fail as the dictionary and resource size changes. NAME does that sound about right?
The alternative floated where 's window size is derived from the size of the dictionary is much more attractive to me. (That is, cap the window size log at , where the spec picks an somewhere in the range .) In such a scheme, where the window size is deterministically derived from the size of the dictionary--a resource the client must already have access to--the client is free to decline to advertise support for for dictionaries which would require the use of a window larger than the client is willing to support. I.e., in resource-constrained environments, Chrome could avoid having to allocate more than 8 MB windows for by not advertising support for on requests where the resolved available dictionary is larger than . Put another way, given that the client will have the tools to make these choices for themselves, I would rather give the client discretion here rather than just bake a flat restriction into the spec.
NAME would you be supportive of updating the zstd command-line tooling to add an option that would set the window to the larger of 8MB or the dictionary size (i.e. match the content-encoding behavior)? My main concern is that it is easy to document and get people to use a fixed command-line for a 8MB window but if it involves a more complicated algorithm then it is likely to break in subtle ways that they may not notice until the resources are bigger than the dictionary (and are bigger than 8MB).
NAME yeah, that would be pretty straightforward. Happy to add that.
If we end up that bundles the dictionary hash with the compressed stream we can make sure the tooling for that file format handles the window size stuff for zstandard automatically (my main concern is around unexpected breakage from different server and client window requirements that only trigger when resources grow). The language will be a bit complex but will probably have to look something like
If I may chime in for some tactical consideration : In an update scenario, it's common for a new resource to be slightly larger than the old one it replaces. The expectation is that we generally tend to add stuff over time. A formulation which caps the window size to the size of the dictionary would introduce problems for all these update cases, which are expected to be the majority of update scenarios. On the other hand, we don't want this increase in size to be completely uncapped, in order to keep control of memory size. And if the new resource is really much larger than the old one, the benefits of delta compression are dubious anyway. So the suggestion is to authorize a window size which is slightly larger than the external dictionary size. As to how much larger exactly, a suggestion here is by +25% (x1.25). I would expect a large majority of update scenarios to fit into this picture, with the new resource being slightly larger than the old one by a few %. For the less common cases where the new ressource is much larger than the old one, Zstandard could still employ the dictionary but only for the beginning of decompression. The compression process will have to select a window size which remains under the specified limit (x1.25 external dictionary size). As a technical insight, note that byte-accurate window size is only a thing when the window is the entire decompressed content. Otherwise, the window size is encoded in a condensed format, which can only represent increments by 1/8th of a power of 2. This is not a problem to determine this "largest window size under the specified limit", but due to limited accuracy, you can already guess what would be the problem is we were to keep the formulation "not more than the larger of 8 MB and the size of the dictionary". Imagine that the external dictionary size is bytes, and the new resource to delta-compress is , aka 1-byte larger than the dictionary. With the initial formulation, since the window size is limited to the size of the dictionary, the compressor cannot use as a window size, because it's 1 byte larger than the limit. So, it will have to find the next best window size under the limit. Due to granularity limitations, this next best window size is 8 MB. So delta compression would proceed fine for the first 8 MB, but for the last 1 MB, data would have to be compressed without the benefit of the dictionary. With the newly proposed limitation to "x1.25 the dictionary size", delta compression would remain active in above scenario during the entire decompression process. And it would remain fully active if the new resource to decompress is 10 MB, 11 MB or 11.24 MB. But if the new resource is 11.25 MB or larger, then the compressor will automatically limit the window size to 11 MB (since it's the largest representable window size under the limit, which, in this particular instance, is ), achieving the wanted memory size control.
Since neither the server not the client may know what the resource size will be when compression starts (say, in a streaming HTML case), won't the 1/8th power of 2 granularity apply even for cases where the resource is the same size (or sometimes smaller) than the dictionary? For resources slightly larger than the old one, the dictionary would still be used for the majority of the resource but the last bit would fall back to regular window-based compression. I agree that having the dictionary available for the whole resource would provide better compression but I'm not sure where to draw the line. Something like a 25% fudge factor can be quite a bit when you start talking about 50 MB+ resources that might be close to where the client wants to draw the line anyway. They may have allowed the dictionary at 50 MB but not allow it's use at all if it needed 62 MB available for a window. A similar case could be made for the non-update case where you have a custom-built dictionary but the resource being compressed exceeds 8 MB. Only the first 8 MB of the resource would be able to use the dictionary. Hopefully at that point the compression window would have most of the stuff that was in the dictionary but the dictionary becoming unavailable is something we should expect for some cases with ZStandard, even if we use a fudge factor. Is there a way to convey the 1/8th power of two granularity on the window size or is it something that the ZStandard libraries will take care of automatically for users (i.e. setting the window size to 11 MB when the caller sets it to 11.24 MB)? I'm wondering how much of this implementors are going to have to deal with directly.
It depends if the compression side (server) is informed upfront of the size of the payload to compress. If not, then indeed you are right, the compressor will have to set a Window Size, and the granularity rule applies. So for example, let's assume the payload to delta-compress is 50 MB, and the reference dictionary is also 50 MB. Let's then assume it implies a window size limit of 50 MB (dictionary size) and the compressor is not informed of the size of the payload. In which case, the compressor will have to set a Window Size, and it will have to settle for the largest representable window size under the limit, which happens to be 48 MB. Therefore, the dictionary won't be used during the compression of the last 2 MB of the payload. With the proposed +25% tolerance rule, the limit would become 62.5 MB, and the largest representable window size under the limit is 60 MB. Now the dictionary remains active during the entire compression of the 50 MB payload. When the data to decompress ends up being smaller than the window size, it's obviously not a problem for the decompression process: the final decompressed size is still properly reported at the end. It's just that the intermediate buffer allocation will be larger than it could have been. Note that uses for the allocation of buffers, so the address space is just reserved, but the physical memory may never be used if it's not needed (i.e. if there is no write action on the later segment part). +25% is more like a suggestion, this can certainly be discussed if some other limit feels better. I liked that it's a (binary) clean factor, and it's larger than , which covers the scenario above, and gives a little bit of breathing room if the newer payload is a bit larger than the older reference one. I see this scenario as being very distinct from delta-compression. For delta-compression, we expect large segments to be identical between the reference and the payload to compress, so we want this ability to access the reference during the entire compression of the payload. The price to pay is that the reference is very specific, and is not designed to offer any serious benefit to any other payload. Custom dictionaries are different: they are supposed to carry the most relevant "commonalities" detected across a set of samples. Presuming a large enough sample set, the common components are not expected to be very large. In such a scenario, the custom dictionary is expected to be (relatively) lightweight (compared to delta-compression), likely in the <= 100 KB range. Consequently, by the time the payload to compress reaches 1 MB, it should already contain all references it needs to compress the rest of the file. I would not expect a large impact of the custom dictionary beyond that point, let alone beyond 8 MB. The plan would be for the library to do this adaptation automatically. The user would not have to know anything about it, though it's better if the user understands that the library selects the closest window size which is <= to the limit, so that they are not surprised if, upon inspection, the final window size is not byte-identical to the set limit. A good API documentation will likely help to reach this objective If by implementors you mean "server side programmers", they would have to know what's the implicit window size limit when a client requests a format (essentially the rule you are currently defining). Then they would set this limit as a compression parameter, and the library will take care of the adaptation into window size.
Yes. (Keeping in mind of course that most traffic in question will presumably be under the 8 MB floor and so none of this complexity will apply.) Right now the zstd library has limited controls about setting detailed window constraints. We'll need to add a new option to give users that fine-grained control. We can do that with an eye towards user simplicity. Frankly, I'm also very open to also adding a flag (and equivalent compression parameter in the library) that would just look at the size of the dictionary (which zstd necessarily has access to) and would configure the window accordingly. In that case, users wouldn't have to do any math at all.
OK, I chaged the structure and language so that they are no longer streams of bytes compressed with a given algorithm but are now specifically and . The references for those named formats have the window restrictions (which for Zstandard includes the 1.25x dictionary requirement). Currently the byte stream itself is still a native Brotli/Zstandard stream, just with specific settings but this also makes it cleaner if we decide to . PR with the update is here: URL
lgtm other than the comment


Changed it to a proper normative reference. For
Merging this. Can always clean it up more if it is still a concern.
For example: Usually, we add something like URLPattern to the references, then that.
Sorry, didn't mean to close the issue. Feel free to close this out if the current language looks OK (also happy to update it if it needs some more tweaking).
Going to close this out now. Thanks for taking the time on this.


The CONNECT-TCP draft makes informative references to other drafts which use "extended CONNECT" (CONNECT-UDP, CONNECT-IP), but does not have a normative reference to the draft which defines this use of the CONNECT method. It references the "extended CONNECT" method without supplying a reference for this term. This draft spells out the required values for each pseudo-header, but omits the requirement to set SETTINGSENABLECONNECT_PROTOCOL. A reference to RFCs 8441/9220 is probably needed, and a reminder that the method in this document can only be used if the server sends this setting first.
That'll do.


This changes the maximum ZStandard compression window to be the larger of 8 MB and the size of the dictionary. This allows for clients to limit the maximum window size to as low as 8 MB if they like (by not advertising support) while also allowing for clients to use larger dictionaries with ZStandard than they would be able to if the global limit was 8 MB.
NAME and NAME could you also take a look and see if the language looks ok?
Context: URL aims to clarify the window size limit for the Content Encoding token as it wasn't clear in RFC8878. Since the Compression Dictionary Transport draft hasn't been published yet, we should also add some window size considerations to it for and to maintain interoperability. I think in Chrome we ended up using . NAME
NAME could you comment on how this impacts Brotli dictionary decode? Is the dictionary independent from the compression window (so we can adopt the same defaults as or do we need to specify something relative to the size of the dictionary? NAME NAME for ZStandard, to allow for delta-encoding large resources, is reasonable or should it be ? We have use cases for delta-encoding wasm applications where the dictionary can easily be upwards of 50 MB.
For Zstandard, the states that the entirety of the dictionary is accessible as long as the input is <= Window Size. Therefore, to give an extreme case, it's possible to have a 50 MB dictionary, and it's entirely accessible even if the Window Size is only 1 KB, but only as long as the data to de/compress is <= 1 KB. As soon as we reach 1 KB + 1, reaching the dictionary is no longer valid. For delta compression, this has consequences. Presuming the reference is ~50 MB, and the updated content is also ~50 MB, one would need a window size of ~50 MB to ensure access into the dictionary during the entire reconstruction process. But if the updated content is just a bit bigger, say ~51 MB, a ~50 MB window size would allow delta operation only during the first ~50 MB, the last ~1 MB would have to be compressed without dictionary. If follows that the size of the dictionary is irrelevant for the Window size. What matters is for how long one wants to access the dictionary while rebuilding the content. And as a general approximation, one wants this dictionary during the entire reconstruction process, so the window size is effectively the size of the content to rebuild. Zstandard has a "windowless" mode which states exactly that : the size of the "window" is the size of the content. For this to work, one has to set the size of the content at the beginning of the compression process . Setting the content size is either an automatic operation when data is provided entirely upfront (), or, when data is provided in small increments (), it must be stated explicitly at the beginning (). will automatically trigger the "windowless" mode when the declared content size is smaller than the set Window size (effectively reducing the memory requirement for the decoder side). I'm not sure if there is another way to trigger it. Note that the decoder will be informed of the Window Size requirement by reading the frame header, and may decide to not proceed if this value is deemed too large. For example, has a default 128 MB limit, and will refuse to decompress payload requiring more that that, unless explicitly authorized to employ more memory (). Just for reference, offers a "large delta compression" mode with the command . It has made us somewhat familiar with some of these issues. In situations where the amount of data to compress and compare is very large, we now prefer to both the dictionary content and the destination file. It has proven useful to reduce memory pressure.
Thanks. That makes things a bit more complicated for the case where we want to require a minimum window size that the server can trust that all clients advertising content-encoding will be able to decode. For the dictionary case it might be cleanest to use the same 128MB limit for window sizes as the default zstd limit (knowing that the client won't allocate that much unless it actually gets a stream that uses it).
Hello. Shared dictionary is placed in the address space between backward reference window and static dictionary. In other words it does not interfere with normal compression window, but using it makes "static dictionary" references more expensive. We decided to order it that way, because shared dictionary works almost the same way as extra window. Sorry for the late response. Best regards, Eugene.
Extra note. As you can see brotli allows backward distances behind window. IIRC, regular (as opposed to large-window) brotli allows distances up to ~56MiB, i.e. larger shared dictionaries would not be useful (== referenced)
Thanks. For the dictionaries to be usable for a broad set of apps (including large wasm bundles), I'm considering spec'ing: Brotli: Regular 16MB window, allowing for delta-compression of resources up to ~50MB. ZStandard: 128MB window, allowing for delta compression of resources up to around that size, depending on where the duplicated bytes are in the updated resource. The main difference with regular content-encoding is on the ZStandard side where the window is 8MB. For platforms that are resource constrained and are not willing to support a 128MB window they would be limited to Brotli (and then decide how large of a dictionary they are willing to keep and use). Another option on the ZStandard side would be to allow for a maximum window that is a multiple of the dictionary size (i.e. 8MB or 2 * dictionary size whichever is larger with a 128MB max). That way the client can have an upper limit that it knows in advance before advertising as being available. That adds complexity to the encoding side though. We have already seen cases where people want to delta-compress resources > 50MB (wasm apps) where they had to use ZStandard instead of Brotli because of the Brotli limits so I don't think a small limit like 8MB for ZStandard would work with this use case.
That massive resources exist is not sufficient justification to build a system that will not be available to some users. I would prefer a far smaller default. If only to encourage the use of smaller resources overall. Defining a separate codepoint for massive windows is possible, say . The scalable option also seems reasonable. Clients can then choose to offer dictionaries when they exceed available resources.
I'm ok using the same 8MB window ( I believe) that we use for for and note that the effectiveness of delta compression for responses > 8MB using may be limited. The scalable option concerns me that people are likely to get it wrong and it won't be obvious and the decompression will start to fail as the dictionary and resource size changes. NAME does that sound about right?
The alternative floated where 's window size is derived from the size of the dictionary is much more attractive to me. (That is, cap the window size log at , where the spec picks an somewhere in the range .) In such a scheme, where the window size is deterministically derived from the size of the dictionary--a resource the client must already have access to--the client is free to decline to advertise support for for dictionaries which would require the use of a window larger than the client is willing to support. I.e., in resource-constrained environments, Chrome could avoid having to allocate more than 8 MB windows for by not advertising support for on requests where the resolved available dictionary is larger than . Put another way, given that the client will have the tools to make these choices for themselves, I would rather give the client discretion here rather than just bake a flat restriction into the spec.
NAME would you be supportive of updating the zstd command-line tooling to add an option that would set the window to the larger of 8MB or the dictionary size (i.e. match the content-encoding behavior)? My main concern is that it is easy to document and get people to use a fixed command-line for a 8MB window but if it involves a more complicated algorithm then it is likely to break in subtle ways that they may not notice until the resources are bigger than the dictionary (and are bigger than 8MB).
NAME yeah, that would be pretty straightforward. Happy to add that.
If we end up that bundles the dictionary hash with the compressed stream we can make sure the tooling for that file format handles the window size stuff for zstandard automatically (my main concern is around unexpected breakage from different server and client window requirements that only trigger when resources grow). The language will be a bit complex but will probably have to look something like
If I may chime in for some tactical consideration : In an update scenario, it's common for a new resource to be slightly larger than the old one it replaces. The expectation is that we generally tend to add stuff over time. A formulation which caps the window size to the size of the dictionary would introduce problems for all these update cases, which are expected to be the majority of update scenarios. On the other hand, we don't want this increase in size to be completely uncapped, in order to keep control of memory size. And if the new resource is really much larger than the old one, the benefits of delta compression are dubious anyway. So the suggestion is to authorize a window size which is slightly larger than the external dictionary size. As to how much larger exactly, a suggestion here is by +25% (x1.25). I would expect a large majority of update scenarios to fit into this picture, with the new resource being slightly larger than the old one by a few %. For the less common cases where the new ressource is much larger than the old one, Zstandard could still employ the dictionary but only for the beginning of decompression. The compression process will have to select a window size which remains under the specified limit (x1.25 external dictionary size). As a technical insight, note that byte-accurate window size is only a thing when the window is the entire decompressed content. Otherwise, the window size is encoded in a condensed format, which can only represent increments by 1/8th of a power of 2. This is not a problem to determine this "largest window size under the specified limit", but due to limited accuracy, you can already guess what would be the problem is we were to keep the formulation "not more than the larger of 8 MB and the size of the dictionary". Imagine that the external dictionary size is bytes, and the new resource to delta-compress is , aka 1-byte larger than the dictionary. With the initial formulation, since the window size is limited to the size of the dictionary, the compressor cannot use as a window size, because it's 1 byte larger than the limit. So, it will have to find the next best window size under the limit. Due to granularity limitations, this next best window size is 8 MB. So delta compression would proceed fine for the first 8 MB, but for the last 1 MB, data would have to be compressed without the benefit of the dictionary. With the newly proposed limitation to "x1.25 the dictionary size", delta compression would remain active in above scenario during the entire decompression process. And it would remain fully active if the new resource to decompress is 10 MB, 11 MB or 11.24 MB. But if the new resource is 11.25 MB or larger, then the compressor will automatically limit the window size to 11 MB (since it's the largest representable window size under the limit, which, in this particular instance, is ), achieving the wanted memory size control.
Since neither the server not the client may know what the resource size will be when compression starts (say, in a streaming HTML case), won't the 1/8th power of 2 granularity apply even for cases where the resource is the same size (or sometimes smaller) than the dictionary? For resources slightly larger than the old one, the dictionary would still be used for the majority of the resource but the last bit would fall back to regular window-based compression. I agree that having the dictionary available for the whole resource would provide better compression but I'm not sure where to draw the line. Something like a 25% fudge factor can be quite a bit when you start talking about 50 MB+ resources that might be close to where the client wants to draw the line anyway. They may have allowed the dictionary at 50 MB but not allow it's use at all if it needed 62 MB available for a window. A similar case could be made for the non-update case where you have a custom-built dictionary but the resource being compressed exceeds 8 MB. Only the first 8 MB of the resource would be able to use the dictionary. Hopefully at that point the compression window would have most of the stuff that was in the dictionary but the dictionary becoming unavailable is something we should expect for some cases with ZStandard, even if we use a fudge factor. Is there a way to convey the 1/8th power of two granularity on the window size or is it something that the ZStandard libraries will take care of automatically for users (i.e. setting the window size to 11 MB when the caller sets it to 11.24 MB)? I'm wondering how much of this implementors are going to have to deal with directly.
It depends if the compression side (server) is informed upfront of the size of the payload to compress. If not, then indeed you are right, the compressor will have to set a Window Size, and the granularity rule applies. So for example, let's assume the payload to delta-compress is 50 MB, and the reference dictionary is also 50 MB. Let's then assume it implies a window size limit of 50 MB (dictionary size) and the compressor is not informed of the size of the payload. In which case, the compressor will have to set a Window Size, and it will have to settle for the largest representable window size under the limit, which happens to be 48 MB. Therefore, the dictionary won't be used during the compression of the last 2 MB of the payload. With the proposed +25% tolerance rule, the limit would become 62.5 MB, and the largest representable window size under the limit is 60 MB. Now the dictionary remains active during the entire compression of the 50 MB payload. When the data to decompress ends up being smaller than the window size, it's obviously not a problem for the decompression process: the final decompressed size is still properly reported at the end. It's just that the intermediate buffer allocation will be larger than it could have been. Note that uses for the allocation of buffers, so the address space is just reserved, but the physical memory may never be used if it's not needed (i.e. if there is no write action on the later segment part). +25% is more like a suggestion, this can certainly be discussed if some other limit feels better. I liked that it's a (binary) clean factor, and it's larger than , which covers the scenario above, and gives a little bit of breathing room if the newer payload is a bit larger than the older reference one. I see this scenario as being very distinct from delta-compression. For delta-compression, we expect large segments to be identical between the reference and the payload to compress, so we want this ability to access the reference during the entire compression of the payload. The price to pay is that the reference is very specific, and is not designed to offer any serious benefit to any other payload. Custom dictionaries are different: they are supposed to carry the most relevant "commonalities" detected across a set of samples. Presuming a large enough sample set, the common components are not expected to be very large. In such a scenario, the custom dictionary is expected to be (relatively) lightweight (compared to delta-compression), likely in the <= 100 KB range. Consequently, by the time the payload to compress reaches 1 MB, it should already contain all references it needs to compress the rest of the file. I would not expect a large impact of the custom dictionary beyond that point, let alone beyond 8 MB. The plan would be for the library to do this adaptation automatically. The user would not have to know anything about it, though it's better if the user understands that the library selects the closest window size which is <= to the limit, so that they are not surprised if, upon inspection, the final window size is not byte-identical to the set limit. A good API documentation will likely help to reach this objective If by implementors you mean "server side programmers", they would have to know what's the implicit window size limit when a client requests a format (essentially the rule you are currently defining). Then they would set this limit as a compression parameter, and the library will take care of the adaptation into window size.
Yes. (Keeping in mind of course that most traffic in question will presumably be under the 8 MB floor and so none of this complexity will apply.) Right now the zstd library has limited controls about setting detailed window constraints. We'll need to add a new option to give users that fine-grained control. We can do that with an eye towards user simplicity. Frankly, I'm also very open to also adding a flag (and equivalent compression parameter in the library) that would just look at the size of the dictionary (which zstd necessarily has access to) and would configure the window accordingly. In that case, users wouldn't have to do any math at all.
OK, I chaged the structure and language so that they are no longer streams of bytes compressed with a given algorithm but are now specifically and . The references for those named formats have the window restrictions (which for Zstandard includes the 1.25x dictionary requirement). Currently the byte stream itself is still a native Brotli/Zstandard stream, just with specific settings but this also makes it cleaner if we decide to . PR with the update is here: URL
lgtm with nits, thank you!LGTM!


This eliminates the response header and moves the hash of the dictionary into the payload of the response. Specifically, it adds 36-byte header in front of the compressed stream which is a 4 or 8-byte signature followed by the 32-byte SHA-256 hash digest. The signature is different for the two different compressions: Dictionary-Compressed Brotli: (0xFF followed by ) Dictionary-Compressed Zstandard: (32-byte skippable Zstandard frame) This also changes the content-encoding names for the different schemes to make it clear that they are a new format and not a raw brotli or Zstandard stream ( and ). From the cli, the stream is equivalent to: or
FWIW, it doesn't look like it would be particularly difficult to use a skippable frame instead of a custom header in the Zstandard case if that's the way we want to go. It's 4-bytes bigger (for the frame size) so it will essentially have an 8-byte "magic number" (since the size will always be the same). It won't help with the creation of the files but if there's benefit to have the existing tooling be able to decode them (without verifying the hash because it will just skip it) we can go that route. On the client side, it feels like it will be cleaner on the implementation to use the same flow for verifying the dictionary hash and magic number for both encodings without having to deal with different sized magic numbers (and always buffer and remove the first 36 bytes) but it's not a significant difference.
Still LGTM
NAME could you please take a look and make sure I got the byte sequence for the skippable frame correct? I tested with the zstd cli and it was able to decode the file with the header without issue when created like this:
Thanks. Updated and added an example with the zstd cli.
This is a discussion we've had several times when defining content codings, but it seems like there is never really a single answer. Should the content coding be self-describing, or can it rely on metadata in fields? The compression dictionary work uses header fields to identify which compression dictionary is in use. Originally, the client would indicate a dictionary and the server would indicate use of that dictionary by choosing the content coding. This made interpreting the body of the response quite challenging in that you needed to have a request in order to make sense of it. More recently, the specification has changed to having the client list available dictionaries, with the server echoing the one it chooses. Both use header fields. There is a third option, which is to embed the dictionary identification (which is a hash of the dictionary) ahead of the compressed content. This has some real advantages: Requests could use delta encoding. There is - of course - a real question about availability of the dictionary, but we have that problem with encrypted content codings and keys. That is a problem we solve for responses with a field. It might be solved using the same technique, combined with the client-initiated content coding advice. Or applications can use their own systems. A field costs a lot more bits. This would not benefit from header compression, so it might be a net loss in some cases, but in many cases it would be a distinct win because 32 bytes of hash in the body of a request is far less than a field name and field value containing a base64 encoding of those same 32 bytes. It also comes with disadvantages: Header compression could remove a lot of the cost in bytes. It might be ever-so-slightly more complex for encoding and decoding to have to split the first 32 bytes off the body of a message.
On the encoding side, the main downside is that the cli tooling for brotli and Zstandard don't currently do the embedding so tooling would have to be added to prepend the hash to the files in a way that isn't standard for either (yet anyway) and for manually decompressing the files. Zstandard has identifiers for the dictionaries when using non-raw dictionaries but both assume that raw dictionaries will be negotiated out of band. Technically it would be a pretty trivial modification for clients and servers that are doing the work, I'm just a bit concerned about the developer experience changes (and whatever needs to be done to get both brotli and Zstandard to understand what amounts to new file formats).
I opened issues with brotli and ZStandard to see if they would consider adding it to the format of their respective file formats. If it's an optional metadata tag that is backwards compatible with existing encoders and decoders I could see it providing quite a bit of value, even in the non-HTTP case of dictionary compression.
There was some discussion in the ZStandard repo about possibly reserving one of the skippable frame magic numbers for embedding a dictionary ID but there's some level of risk for collision with people who may be using those frames for watermarking or other application-specific use cases. As best as I can tell, the brotli stream format doesn't have a similar frame capability for metadata or application-specific data. We could create a container format that held the dictionary ID and stream (header basically, not unlike zip vs deflate) but that feels like a fairly large effort and the tooling would have to catch up to make it easy for developers to work with. At this point I'm hesitant to recommend adding anything to the payload itself that the existing brotli and zstd tooling can't process. Being able to create, fetch and test the raw files is quite useful for the developer workflow and for debugging deployments. Would it make sense to allow for future encoding formats to include a dictionary ID in the file itself and make the header optional in those cases (and make the embedded ID authoritative)? I'm not sure if that would make sense in this draft since this one is limited to the 2 existing encodings and it would be addressed in a new draft when the new encodings are created or if it makes sense to allow for it here without requiring that and explicitly require embedded ID's.
I don't think that you want to change the zstd or brotli format, only the content coding. That is, something like this: This does partly work against the idea that you might have a bunch of files that contain compressed versions of content. You can't just say . And you can't just decompress without some processing. What you get in exchange is a message payload with the advantages I described.
I do think we need to define a file format for it if we go down this path to ease adoption and it should probably have a magic signature at the beginning. Maybe something like gzip is to deflate but with a simple 3-byte signature followed by the hash followed by the stream data. Assuming we create a cli tool to do all of the work for compressing/decompressing them, I'll ping several of the current origin trial participants to see how it will fit into their workflow. Something like: Dictionary-compressed file/stream format with a 3-byte signature indicating hash and compression type, followed by dictionary hash, followed by raw compressed stream. Something like for Dictionary-Compressed Brotli with Sha-256 hash and for Dictionary-Compressed Zstandard with Sha-256 hash. Create a cli script (cross-platform, available in npm, bun, apt, etc) that creates the files and (optionally) calls out to brotli and zstd to do the compression. We could include any special ZStandard compression window handling in the format spec as well as the cli tool. I'm assuming something like this would be better off as it's own draft that this references or do you think it makes sense to define it here? I agree there are significant benefits to having the hash paired directly with the resource. I just want to be careful to make sure whatever we do fits in well with the developer workflow as well.
Adding the header would add some work to typical CI workflow. At least in my case, the diff file stream is created using the cli using the options. Adding the header would require creating a new file with the header and add the generated stream. If this can be done in a published script it would simplify the logic (FYI - URL currently does yet not support brotli bindings, see URL) but not a requirement IMO. Do I understand correctly that with this idea implemented the header would not be necessary anymore? That would simplify some CDN configurations which currently needs to add it. Related to this - what in this case would be header value? current spec is it should be which I understand as brotli dictionary stream. If we add the header, how would a client be able to determine which one it is? Should we add another content encoding for this?
Yes, this would eliminate the need for the header. The would be and the definition of would be changed to reference the stream format of a brotli stream with the header prefix. There would be no content-encoding support for a bare brotli dictionary-compressed stream. At that point, maybe renaming them to and for the content encodings would also make things clearer.
I have a question. If we can use the new and in the header, why do we need to have the "3-byte signature indicating hash and compression type" in the response body?
It's technically not required but it makes it safer and easier to operate on the files outside of the http case. For example, here's some discussion on the brotli issue tracker from 2016 asking for the same: URL
I don't have a problem with doing that in this document, if that is the way people want to go. I personally don't think that a new format is needed because this is a content-encoding, not a media type. But if a media type (and tooling) helps with the deployment of the content-encoding, then maybe that is the right thing to do. Either way, I don't think that you should make that decision (it's a significant one) without having a broader discussion than just this issue.
NAME - could you help me understand this one? (mainly, why wouldn't we be able to apply delta encoding in the absence of an in-body hash?) This will definitely add complexity (e.g. the need to define a new file format that would wrap dictionaries, along with required tooling). It's not currently clear to me what the advantage of this would be. Do we predict the use of this HTTP mechanism outside of an HTTP flow? If so, concrete examples would be helpful.
NAME is the complexity you are concerned about limited to tooling and spec process or do you also see it as being more complex after we are at a good state with tooling? Assuming the brotli and Zstandard libs and cli tools have been updated to add a flag for "embedded dictionary hash" format streams, does one start to look better? For me, the hash being embedded in the stream/file removes fragility from the system. It blocks the decode of the file with the wrong dictionary and enables rebuilding of the metadata that maps dictionaries to compressed files if, for some reason, that metadata got lost (file names truncated, etc). It also feels like it simplifies the serving path a bit, bringing us back to "serve this file" based on the request header vs "serve this file and add this http response header". On the size side of things, I expect it will likely be a wash. Delta-compressed responses will be a few bytes smaller because the header (name and value) is larger than the file header (10's of bytes, not huge by any means). In the dynamic resource case where multiple responses will re-use the same dictionary, the header can be compressed away with HPACK/QPACK, making the header case a bit smaller. I don't think it's a big change in complexity/fragility one way or the other but it does feel like there are fewer moving pieces once we get to a place where the tooling is taken care of to have the file contents themselves specify the dictionary they were compressed with. The need for tooling changes would delay adoption a little bit so it's not a free decision but I want to make sure we don't sacrifice future use cases and simplicity for an easier launch.
I'm not sure that I see the tooling process as critical, relative to the robustness. And performance will be a wash (though I tend to view the first interaction as more important than repeated interactions). For tooling, if this content is produced programmatically, then there should be no issue. Integrated tooling can do anything. If content is produced and stored in files, then seems fine. That's a one-way operation generally. I don't see the definition of new media types to be necessary as part of that. I don't see these files being used outside of HTTP, ever. Maybe you could teach the command line decompression tools to recognize the format and complain in a useful way, but that's about the extent of the work I'd undertake there. You could do as NAME suggests as well, which would be even more useful, but given the usage context, that's of pretty narrow applicability.
I'm mostly concerned about official tooling and the latency of getting them to where developers need them to be. Compression dictionaries already require jumping through some , due to latency between brotli releases and adoption by the different package managers. At the same time, if y'all feel strongly about this, this extra initial complexity won't be a deal breaker.
I think there are enough robustness benefits that it is worth some short-term pain that hopefully we will all forget about in a few years. On the header side of things, how do you all feel with respect to a bare 32-byte hash vs a 35-byte header with a 3-byte signature followed by the hash (or a 3-byte signature followed by a 1-byte header size followed by the hash to allow for changes as well as 4-byte alignment)? It's possible I'm mentally stuck in the old days of sniffing content but since the hash can literally be any value, including accidentally looking like something else, I like the explicit nature of a at the beginning of the stream. It essentially becomes (or whatever the signature is) so it doesn't add significantly to the complexity but it does add 3 bytes, potentially unnecessarily.
+1 to adding a 3 byte magic signature if that's the route we're taking.
I'm ambivalent on the added signature, so I'll defer to others. I can see how that might help a command-line tool more easily distinguish between this and a genuine brotli-/zstd- compressed file and do the right thing with it. On the other hand, it's 3 more bytes and - if the formats themselves have a magic sequence - the same tools could equally skip 32 bytes and check for their magic there.
I think I am having the same response as you all in the opposite direction, where to me it feels preferable to make it HTTP's problem so that my layer doesn't have to deal with additional complexity. :smiley: But if I overcome that bias and accept that it would be nice to avoid an additional HTTP header, here's what I think: If we used a Zstd skippable frame, that would change the stream overhead to 8 bytes (4 byte magic + 4 byte length) + 32 byte hash. But it would mean that existing binaries would be able to decode the frames and avoid ecosystem fragmentation. That's a lot more attractive to me than a new format along the lines of which (1) existing tools won't understand and (2) isn't a format I'd feel comfortable auto-detecting by default in the CLI because isn't all that unlikely a beginning of a string in the way that is (non-ASCII, invalid UTF-8). And I've thought about it more and I'm actually not concerned about colliding a skippable frame type with someone else's existing use case. It would be more of a problem if we were trying to spec a universal solution, but if we scope this to just this content-encoding, then we're free to reserve whatever code points we want and attach whatever semantics we want.
Thanks. I guess the main question I have is if there would be benefits to Zstandard itself for the files to be self-contained with the identification of the dictionary that they were encoded with or the possibility of mismatching dictionaries on compression and decompression (or being able to find the matching dictionary given just the compressed file) are issues that are HTTP-specific. I'm fine with specifying that the encoded files carry the dictionary hash (before the compressed stream data) and having different ways for Zstandard and Brotli to do that actual embedding. That said, the tooling gets more complicated on the encode and decode side to generate the frame and insert it in the correct place in the compressed file and at decompression time, that makes the client more format-aware, having to parse more of the stream to extract the dictionary and then re-send the full stream through the decoder (at least until the decoder library becomes aware of embedded dictionary hash).
NAME Is this really what you want in this case? The decoder needs to know where to find the dictionary, so wouldn't you want this to be a breaking change to the format such that a decoder that has a dictionary is fine and a decoder that doesn't knows to go get one. ... And - importantly - an older decoder will abort. (I confess that I don't know what the zstd frame extension model is and didn't check.)
I don't think we're contemplating a model where Zstd can ingest a frame and figure out on its own where to find the dictionary it needs and then load it and use it. I expect that the enclosing application will parse this header, get the hash, and then find the dictionary and provide it to Zstd. The advantage to using the skippable frame is that then you can provide the whole input to Zstd (including existing versions) and it will work rather than have to pull the header off.
One thing I realized now - by assuming that the hash length is 32 bytes, we're assuming the hash would remain SHA-256 forever. That might be how things play out, but it that we'd need to change hashes at some point. If we were to do that, having a fixed length hash as part of the format would make things more complex.
Having a fixed-length hash (or fixed hash) as part of a content coding is perfectly fine. If there is a need to update hashes, it is easy to define a new content coding.
NAME the doesn't use skippable frames and uses the same custom header for Brotli and Zstandard (with different magic numbers). I can switch to using a skippable frame instead (which effectively just becomes an 8-byte magic number since the frame length is always the same) but I'm wondering if it makes sense and is worth adding 4 bytes. It won't help in creating the files so it's just for decode time and the main benefit you get is that you can use the existing zstd cli and libraries to decode the stream without skipping the header but those also won't verify the dictionary hash, they will just skip over it. That might not be a problem but part of the decode process will be to fail the request if the hashes don't match.
In talking to the Brotli team, it looks like Brotli already and validates it during decode to select the dictionary to use. It uses a "256-bit Highwayhash checksum" so we can't use the hash to lookup a dictionary indexed by SHA-256 but we can use it to guarantee the decompression doesn't use a different dictionary (and the existing libraries and cli tools already use it). NAME when you were concerned about the , was it for both lookup and validation or just validation? I'm just wondering if we can use the existing brotli streams as they already exist or if we should still add a header to support locating the dictionary by SHA-256 hash.
There are a few places in HTTP where the interpretation of a response depends on something in the request. Everyone one of those turns out to be awful for generic processing of responses. That's my main reasoning, so I'd say both: lookup first, then validation. That is, my expectation is not that the client has a singular candidate dictionary, or that it commits to one in particular. So in the original design, when you had the client pick one, that didn't seem like a great idea to me. For validation, Highwayhash (NIH much?) doesn't appear to be pre-image resistant, so I'd be concerned if we were relying on the pre-image resistance properties of SHA-2. Can we be confident that this is not now subject to potential ployglot attacks if we relied on that hash alone? That is, could two clients think that they have the same resource, but do not?
I wouldn't be comfortable switching hashes given the wider and proven use of SHA-256 and agree on the lookup case to allow content-encoding with different and multiple dictionary negotiations. Looks like a separate header is still the cleanest so the main remaining question is if we use the same style of header for both or a skippable frame for Zstandard. I also need to update the brotli citations to point to the shared dictionary draft format instead of the original brotli format. On Sun, May 12, 2024 at 7:58 PM Martin Thomson NAME wrote: >
Sorry for the late comment. It seems to me that conveying this information in the content eases the integration with Signatures and Digest. It is not clear to me if there are still possible cases where the response does not contain all the information required for processing.
Thanks
Looks like you just need to update the PR summary to reflect this change.LGTMOverall, I like this. I want to flag for others that it doesn't use the skippable frame concept.


Adds language to the to not send the dictionary-aware encodings when a dictionary is not available (or when one is available but not being used). The dictionary-based encodings don't work without a dictionary (they contain a header that has the dictionary hash) and should not be advertised when they are not really available.
Thanks


This simplifies the CORS processing language in the doc, leaving all of the client-side readability processing to fetch and clearly spelling out the server responsibilities.
The new language is much better, but I think it still misses the point. We don't need to be shy about why these requirements are in the spec and under what circumstances they are necessary. I suggest starting with an explanation:
Thanks. I pulled in your explanation and tweaked the language to make it clear that precautions should be considered, one of which may be to not use dictionary-based content encoding. I like the new version a lot better as it allows for cacheable, public data to still use dictionary compression even if it is fetched in a cross-origin context and isn't client-readable. I'll have to be careful with the fetch language as, currently, the client will fail a cross-origin non-readable response in case the server hasn't applied relevant protections. It may be reasonable to relax that and allow for the client to process it and put the burden on the server but it might also let a weak configuration go unnoticed (though since attacker-controlled dictionary content isn't particularly practical in this model, maybe it's not really a concern).
For the existing draft, there is a lot of unnecessary confusion regarding features of fetch, like CORS, that don't make any sense from a security perspective. That's not what CORS is capable of covering, nor how it is implemented in practice, so reusing it doesn't make any sense. cc NAME Moved from private ID
CORS covers privacy from a browser perspective as far as the readability of responses relative to the origin of the containing document which is exactly the context that it is needed for here. The concern that it takes care of is to make sure that responses that shouldn't be readable from the document context of the client can't be exposed to oracle timing attacks (because there won't be any client-opaque responses). HTTP itself doesn't really have the same document framing context and need for protecting read access of individual responses on a shared connection by clients running in different document contexts.
From NAME I still don't see how CORS applies to the draft. If the origin is choosing both the available dictionaries and whether or not to apply them to a given response, why is the context of a request relevant to the nature or credibility of the compressed response? Neither one is controlled by the context. The origin server isn't going to attack itself. The client might choose not to advertise an available dictionary based on the context, but I don't see how ignoring a server's valid response makes a difference. I assume the content of the response after decompressions would still be restricted by CORS, but that should already be defined by fetch; I wouldn't expect this document to specify it.
The concern raised by the security team is that a client could abuse dictionary compression and a timing attack to expose the contents of a CORS-opaque response. The contents themselves would already be prevented from being exposed directly but there's a theoretical risk that an origin serving private data through dictionary compression could expose that data unexpectedly in a cross-origin timing attack. I don't know enough about the practicality of timing-based oracle attacks using dictionary compression and the level of control that an attacker would need to have on either the payload of the response or the payload of the dictionary but in an abundance of caution we are recommending that servers avoid the problem entirely and don't use dictionaries in cross-origin opaque responses (even if it would be blocked from being processed in the client).
The shared-brotli draft has a fairly good write-up on the CRIME attack concerns which is what the CORS mitigations prevent (by making sure there is no mix of public and private data and only allowing for public data): URL Most of the CORS protections are on the client side and not something that needs to be exposed to HTTP or for origins to be concerned about. If a server never mixes public and private data or doesn't talk to a client that needs to partition that kind of data (i.e. not serving to a browser) then it can also ignore the one remaining case called out in the spec. If there's a chance that public/private data will me mixed AND the server is serving content to a browser (or other client that implements CORS) then there is a risk of a timing attack being able to reveal the private content if the crossorigin not-allowed case uses dictionary compression.
Closing this as the need for CORS for the client protection should be well documented at this point (and mostly applies to the WHATWG side of things in fetch processing but there is one case here where people should at least be aware of a possible attack vector if not handled).
I have no problem with this being a requirement for the fetch implementation (or for script-enabled browser implementations in general). However, I still don't understand how the requirements in 9.3.2.* will pass muster for the IESG and RFC Ed reviews without a normative reference to the Fetch spec. A link to a web page isn't enough. Personally, I think these details should be moved to the fetch spec (where the right people will read it). The bits needed for the IETF are covered in 9.3.1 and the first two paragraphs of 9.3.2: and then everything else in 9.3 (and below) can be a replaced by a non-normative reference to fetch. Otherwise, this should be a normative reference and the several links to fetch replaced with an appropriate [FETCH] entry in 11.1.
To be clear, I don't object to moving forward with this as is, but I don't believe will be allowed as is, unless the chairs have some extra special hand-waving skills I haven't seen yet.
I almost take that as a challenge...
There is processing model stuff that is browser-specific that will be in fetch but the parts in 9.3.2.1 and 9.3.2.2 are the things that are required from any HTTP server that might be used to serve sensitive information to a browser. I stripped out as much as I could but there is a timing attack that can be performed in fetching cross-origin content that would allow a client to "read" the response even if it doesn't have access to it on the client side and the only way to prevent it entirely is to not serve a response in the first place. By making it only a requirement when the CORS headers are present, it allows servers that never talk to browsers to not have to consider it but it really does need to be considered carefully for any server that might serve responses to a browser (or anything else that implements cross-origin readability restrictions). Making the protection non-normative would make the privacy protections optional and risk servers not being aware or necessarily implementing it (and it is the same side-channel timing attack that killed SDCH). If we're comfortable making that change then I can make the fetch spec parts authoritative about the CORS restrictions on both clients and servers but it would probably have to be a normative reference.
Oh, I see, the server description contains things a server MUST do (or not do) based on the values received in those fields when they indicate a cross-origin request. I was reading them as things a client would verify after receipt of the Response, since we usually don't rely on a bad client sending "I'm good" information in request header fields. [BTW, this is why the main HTTP specs always target requirements to a role instead of making them neutral requirements on a message.] In that case, these are definitely normative requirements, similar to conditional requests. HTTP servers don't normally parse CORS header fields, since pre-flights and such are handled by back-end applications (not httpd).
I'll make another pass over the CORS language tomorrow to make it clearer and make sure it only covers the parts that servers need to handle for fetch-compliant clients. The client bits can and should be in fetch.


Updates some out of date portions of the IANA considerations section. See for more info
NAME Ping
NAME Ping
Some aspects of the IANA considerations section are outdated and should be updated: The “Cookie” and “Set-Cookie” registrations from RFC 6265 have been moved from the “Permanent Message Header Field Names” registry to the “HTTP Field Name Registry” Update Sections 9.1 (Cookie) and 9.2 (Set-Cookie)The word "Registry" should be omitted from registry names. Update Section 9.3 (Cookie Attribute Registry)
LGTM. Apologies I missed this.


HTTPSEM has been published as RFC9110, update all reference to reflect this.
NAME Can you PTAL? I spot checked that existing section #s still matched up, but I would appreciate another look.
The sections should be stable, see . I'd also change "HTTPSEM" to just "HTTP2".
Sorry, not sure I understand.
Oops. I meant to say: I'd also change "HTTPSEM" to just "HTTP". See URL
Thanks, the latest commit should fix that.
Still LGTM. Thanks!


Adds suggestions for how servers could fulfill the requirement Adds suggestion that servers be mindful of the number and size of the cookies they set to avoid hitting their own header length limits. Also mention that UA should consider splitting longer fields.
If that's the case, I'd recommend punting on getting this in 6265bis - which has had a goal of documenting reality.
It is not unusual for servers to reject HTTP requests that contain individual headers longer than a certain limit. If a client sends a header longer than around 8-9K bytes, the risk is quite large that it will get a 40x response back which effectively blocks the client from continuing what it is set out to do. Because of the rather complicated results of such a "too long" cookie header, I think it could be worth mentioning this in the spec. For newcomers in cookie land, this is perhaps not completely obvious. The spec currently mentions size-limits per cookie, but a server can easily send a few large cookies and have the combined size of them outgrow 8K. I would perhaps suggest wording that says a client SHOULD keep its lines shorter than NNN bytes and servers SHOULD accept cookie lines up to at least NNN bytes. (in curl we have the limit set to 8190 bytes) HTTP related specs rarely speak of this kind of size limits, but I know my implementation could have avoided some issues if this real-world limit had been mentioned.
Seems like a worthwhile warning to me.
RFC 6265 had a requirement that a client MUST generate a single cookie header. HTTP/2 and HTTP/3 allow "cookie crumbling" that permits multiple Cookie headers. 6265bis says in URL What is tolerance really? Why are we stating MUST requirement that are immediately invalidated by two of the versions of HTTP in wide usage? Section 5 is huge, what requirement is the text in section 4.2 forward referencing to? I think there's some room for improvement in this area. An HTTP/2 or HTTP/3 is being instructed as a MUST requirement in the relevant RFCs to combine multiple cookie headers into one e.g. URL but if the server MUST be tolerant to multiple Cookies, maybe we can relax the HTTP/2 or HTTP/3 requirements?
This was the output of the discussion in URL
Thanks Mike, I missed that discussion first time around. My question comes about because of some HTTP/2 code I looked at that simply implements HTTP/2 and so combines the multiple fields back into one. But as NAME and NAME alluded to their comments on , the splitting could be done for some reason thst isn't purely about H2 or H3 compression optimization. It seems there are actually use cases where keeping the split is useful, and not just in a private deployment. Of course relaxing things more might have unintended consequences. I find the current text too terse to be useful. This is a cookie spec, it seems like the right place to ba adding commentary about how to handle cookies across various version and conversions. Especially some anecdotal examples about why some clients actually do it.
Mentioning why a server should be tolerant, and what shape that could take, seems reasonable to me. Especially given how nicely it dovetails with . What forward reference are you referring to? ?
In that mention, I think it's worth including that re-combining the Cookie lines before passing to the next layer is a form of tolerance from the client's perspective.
Some more text sounds good and I'm happy to review. In Section 4.2.1:
Still LGTM, thanks!


In 4.2, the parameter is "a byte sequence that contains the public key", and the encoding of the public key is taken from Section 3.1. When reading this, I was initially unclear whether was only the key or the entire structure defined in Section 3.1 or another structure, particularly given the use of the word "contains." I might clarify that the parameter is the public key, and perhaps point from both 3.1 and 4.2 to the same place for how to get the bytes which are being base64url-encoded here. (Make a 3.1.1?) (Marking editorial, as I don't think I'm asking for a technical change here, just clarification of how this already works.)
What you're describing is indeed what we intended. I'm write up an editorial PR to clarify this.


The five authentication parameters are offset in various ways throughout the document. Examples: using the Parameter (Section 3.3) using the v Parameter (Section 3.2) The REQUIRED "p" (proof) parameter (Section 4.3) Additionally, the word "directive" is used instead of parameter in Section 4.4. It might be advisable to align the document on always using one of these, with a possible difference where the parameter is being defined.
Agreed. I'll write a PR that standardizes on parameter and not directive "p" when defining a new parameter when referencing the parameter


In Section 8: Given that the exporter can be produced at any point in the lifetime of a TLS connection and that TLS connections can remain open for a very long time, it's not clear that this guarantees freshness in the same way that a nonce does. It might be clearer to state that it guarantees the authenticator is no older than the beginning of the current TLS connection. (I presume that a server can use GOAWAY to get rid of connections it considers unacceptably old.)


Summarizes all the major changes from 6265bis, replacing the existing draft-by-draft change log.
While there are still a couple open issues that may/may not change this PR, I'd still appreciate an initial look. Thanks!
NAME This is ready for review, I don't believe the pending PR needs to be added to this changelog
Looks generally reasonable to me. Maybe drop the IANA changes summary (too detailed), and try to lead with the important changes (e.g., drop the errata down to last)?
It'd be nice if there were an appendix summarising the changes. We do this for most updates.
Sorry for being a bit dense but what is URL in comparison to that? :)
What's needed is a summary of changes from 6265, not for draft-by-draft. And that needs to stay in the spec upon publication.
NAME I see, thanks!
Is there a rule of thumb for which changes should be included? "Major" changes? Any normative changes? Something in between? Also, do you know of any existing RFCs that are a good example?
Major changes definitely, and any significant normative changes. See eg URL
Catching up on things after being out sick for a ~week. This change LGTM. I think the level of detail is appropriate, and I didn't see anything in the line-by-line changelog that jumped out at me as being big enough to consider missing.


defines a framework for problem types, which can be used as a standardized response format when errors occurs. With resumable uploads, we have a few cases where such problem types could be useful, but usage remains entirely optional. The text in this PR has been inspired by similar sections from .
Thank you very much for quick feedback, Lucas!


minimalist attempt at a fix for
Thanks for the review NAME I've incorporated all your suggestions.
Thanks NAME and NAME I dare say it's okay merge? (thinking about the I-D submission cutoff in a few hours but admittedly don't know what else could/should go into -15 or if publishing that today is even on the table)
I don't believe this example given in particularity well reflects the situation that precipitated the behavior described in : It's only an explanatory example so arguably doesn't even really matter. But RFCs are forever and they are often cited so it'd be nice to have this bit of the historical narrative better reflect the realities at the time (I say this mostly from perspective of having experienced some of the difficulties caused by the introduction of SameSite cookies and their default enforcement mode and, to this day, continuing to have challenging conversations on the topic). As best I can recall, login type flows were the main driver for the "Lax-allowing-unsafe" default mode but the need manifested itself more in the later part of the flow where the authenticating site (IDP) caused the browser to make a POST request callback to the site requesting authentication (RP). At the start of the flow, prior to sending the user to the IDP, the RP would typically set a cookie with or referencing some transaction specific state and use that info during processing of the callback for validation of the token provided by the IDP. Cookies with Lax as the default enforcement mode would not be sent with the callback POST request and the whole login flow would fail. The "Lax-allowing-unsafe" default mode (aka "Lax + POST mitigation") allowed those flows to not break for a while and gave RPs (and their vendors, frameworks, etc.) a little grace period to update to explicitly designate those cookies as SameSite=None. Could we augment or replace the example described in to account for all that? The cookie age stuff in , I think, makes more sense in the context of the above too. I'm (obviously) having trouble articulating some of this so including some "helpful links" here too, which may or may not describe some of this better. Chromium called it the "Lax + POST mitigation" where the cookie would have a CSRF token: URL This explainer by some Googlers describes some of this in Top-Level Cross-Site POST Requests: URL and links to this payment flow which is very similar to the login flow: URL The question in this issue on kinda related stuff came back to my attention very recently, which is what led me to looking at these parts of RFC6265bis again. Which maybe explains some of why I'm putting in this annoying issue near the very end of WGLC. Or explains the timing anyway. URL
To clarify, You'd like the example to be one which is closer to a real world case? Namely FedCM (guessing from your use of "RP", "IDP", and the issue on the repo). I'd like to avoid getting too specific and in the weeds but there's probably a way to generalize the FedCM case. Contextualizing the cookie age also seems like a positive to me, its relevance isn't super clear otherwise.
Yes. Apologies if that wasn't clear from my rambling above. And to try and be more clear, I'd like the example to be one which is closer to the real world cases that gave rise to the Lax-allowing-unsafe mode. No, namely SAML with the authentication response delivered with the POST binding. Or OpenID Connect using the where the '' is needed to validate the returned ID token. Agreed about not getting too specific and in the weeds but think the SAML/OIDC cases could be generalized too. Although I tried to do that in the issue description here and didn't seem to be too successful in conveying the concept. In my mind it makes a lot more sense in the context of a short term cookie used for transaction specific state and validation. And doesn't make a lot of sense in the context of "cookie[s] with login information". I guess my hope is that it would just be better contextualized by way of using an example that's more reflective of the historic occurrence. But more contextualizing, if needed/appropriate, would certainly be okay by me too.
NAME could you propose a diff to the paragraph: Or is it sufficient to just tweak the first sentence of that paragraph? For someone who is not super familiar with SAML authentication flows ( , it's me), what exists in the draft text today feels like a fuzzier higher-level description than: It's entirely possible that's the only pattern that motivated Lax-allowing-unsafe, but maybe not - the web is messy. Maybe a compromise here is to tweak the text to cover your example generally (as well as other examples sitauations to ~POST w/ a cookie expecting some login information), and go super precise in the commit message. Just a thought.
It'd be a bit more involved than just the first sentence but I'm happy to take a stab at reworking that paragraph and maybe a few other very minor related things, if the authors/WG are amenable to the idea? Not to be expected :) The OpenID Connect flow is probably more salient but familiarity there isn't expected either. The important distinction (to me anyway) is where the cookie in question would be sent (or not sent), the associated failure mode, and the effectiveness of the mitigating default mode. With what exists in the draft text today, the cookie is holding/referencing login state that's sent to the site that will authenticate the user (maybe based on an existing session identified by that cookie). If the cookie isn't sent (due to default of Lax and a x-site POST style redirect), the failure mode will usually be the user needing to enter their login credentials again. Which is annoying but recoverable. Also the Lax-allowing-unsafe applicability to "cookies which were created recently [~2 mins]" doesn't help in many many cases for cookie that's used for login state due to such cookies being relatively long lived (not created recently) and/or the user not having even visiting the site recently. With the pattern I'm (trying) to describe, the cookie is holding transactional state that's sent back to the site requesting authentication after user authentication. If the cookie isn't sent (due to default of Lax and a x-site POST style redirect back to the originating site), the failure mode will usually be an unrecoverable error for the whole login flow. The Lax-allowing-unsafe applicability to "cookies which were created recently [~2 mins]" is quite helpful in this case because that transactional state specific cookie will typically be set right at the beginning of the whole login flow and only needs to live as long as it takes for the user to complete the login journey. Oh yes, of course, it's very messy. I'm sure it wasn't the only pattern that motivated Lax-allowing-unsafe but I'm almost as sure that it was the primary pattern that motivated it. Let me see if I can tweak the text to cover things generally (without screwing it up) and the commit message can reference this issue to link back for more info?
Hi NAME Have you had any progress?
I'm sorry NAME probably my fault in the way I left the prior but I wasn't sure if I was supposed to be working on a PR or was waiting for an okay from you/WG/etc to do so. And in the intervening time my attention for it was sort of overtaken by events. Apologies again and I'll try and do something soon.
URL is a minimalist attempt at some adjusted text for this
LGTM, thanks! I'll wait for NAME to take another look before mergingCatching up on things after being out sick: this still LGTM. :)


This is a follow-up to , which defined a problem type for when the server rejects modification for an completed upload. However, the draft didn't specify that such modifications are not allowed. This PR adds this requirement. The section for offset retrieval already includes a definition for when an upload is complete, so this is not repeated in this PR's changes:


The draft says there are 4 upgrade tokens, but a fifth one has since been added: connect-ip. It already has text to avoid this issue so it should be a short section.