This PR is one potential way to It simplifies route exchanges by going with NAME 's suggestion of sending the entire routing table each time something changes. It could be somewhat inefficient in some cases, but it becomes easier to reason about than what's currently in the draft when overlapping prefixes are involved.
This is interesting, but I'd prefer to wait to do this until after we post a -00 version. NAME is that OK with you?
I'd prefer we figure this out before -00, because this would be a pretty significant non-compatible change and I'd like to implement -00
NAME you nearly convinced me in the call that we rather want to announce ranges separately but I forgot why. Can you outline your reasoning again?
NAME my argument was that we shouldn't have two different ways to set routes, one that gives a table and one that gives a single route. If we go to only have a table, then it's fine.
I know saw the discussion in and support this approach. Also if we agree on this approach, I also think we should merge it before we submit -00
NAME I think we're ready to merge this PR
As discussed in , turning a sequence of ROUTEADVERTISEMENT and ROUTEWITHDRAWAL capsules into a functional routing table isn't as simple as initially expected. This is because two separate capsules can describe prefixes that are distinct but overlap. We should simplify the design or better explain the caveats.
I think this looks good enough to include in the -00.


Instead of just an address / hostname, could we ask to open a tunnel to a prefix?
Tommy, can you please clarify the reasoning why this functionality is useful. Specifying the target as a subnet would to my understanding compared to a general IP tunneling address just restrict that the client will never try to send IP packets to any IP addresses outside of this subnet. And thus potentially enable serving multiple masque clients using restricted targets to share the source IP address. Is that the intention? Or is the intention here to have the masque server potentially redirect to another instance based on the target in the request?
This came up in a discussion—if we let you ask to connect to a specific hostname or address instead of all addresses (as we do today), we wondered if there was any need to connect to a subnet of addresses. This would be for connecting to an IPv6 /64, which is unique enough for a device, and doesn't require needing to assign an IP address exclusively to the client that is issuing the request.
I'm supportive of adding this if we can come up with an encoding that's not a pain to implement. For example slash with prefix length seems fine to me - the slash would be percent-encoded just like the IPv6 address colons are.
In relation to my just raised issue on source address validation if we add this functionality, I would think that the proxy would need to do destination or source address validation depending on direction of traffic to enforce the signalled aspect.


I think there are two option here, 1) close/deny the connect request, and 2) use streams instead. I think we should consider both option, especially given stream are just there to use.
Isn't this a dup of ? That's fixed with . I guess the text could be generalized to talk about more than just IPv6 MTU issues.
No, this aspect on what you do when the MTU required is not present is not resolved by that PR. I would note that closing gives the endpoints one experience that is likely easier to handle in implementation and do avoid the issue with reordering and the varying packet treatment that moving to stream would result.
I would suggest adding a sentence saying that if an endpoint realizes that the IPv6 tunnel MTU goes lower than 1280, then it MUST close the stream.
IETF 114: Note that if something happens and your MTU gets smaller, you may need to destroy that link. Potentially make this option a SHOULD do PMTUD, otherwise you could end up in a situation where you're not a compliant IPv6 tunnel. Proposal:
Even if PR was an improvement to the description of the IPv6 minimal link MTU issue the text still leaves an open issue in that it doesn't make clear which of the endpoints that are responsible for HTTP intermediaries or if is truly both which has another implication. So a Connect-IP endpoint will be aware of the HTTP datagram supported MTU and if its HTTP/3 connection can support sending at least 1280 bytes of IPv6 packets without fragmentation. However, if there is two or more HTTP intermediaries in between the client and the server then there are HTTP intermediaries that might have an HTTP/3 connection which has less than 1280 bytes MTU for HTTP datagram payloads in-between the client and server. Thus, arises the issue of how to detect this. PR states: if HTTP intermediaries are in use, CONNECT-IP endpoints can enter in an out of band agreement with the intermediaries to ensure that endpoints and intermediaries pad QUIC INITIAL packets. From my perspective the big question is how a client or a server knows that there are intermediaries, or the most important aspect, that there are no unknown HTTP intermediaries. To my understanding both the client and the server can potentially bring in HTTP intermediaries unknown to the other part. Thus resulting in a HTTP intermediary to intermediary connection where neither endpoint knows if the requirement is fulfilled. Thus my interpretation is that the only way to be sure that it works it to actually execute on the third bullet in the PR: CONNECT-IP endpoints can also send ICMPv6 echo requests with 1232 bytes of data to ascertain the link MTU and tear down the tunnel if they do not receive a response. Unless endpoints have an out of band means of guaranteeing that one of the two previous techniques is sufficient, they MUST use this method. I would also note that this only verifies from the acting endpoint towards its peer, not the reverse path. For that to be verified the peer need to send ICMP probes the other direction. Thus, my conclusion is that Connect-IP still don't ensure that the minimal MTU is fulfilled.
Isn't the quick-fix to require that the ICMP reply also be 1232 byte sized?
Normally I would say now, as you want to ensure that the reply makes it back that shouldn't be padded. Also if the response is anything else than echo reply you also should not pad. However, I think the implication is the same either way. Both masque client and server need to send ICMP or modify its ICMP processing. Given that I would think having each endpoint doing independent processing of its forward path might actually be simpler and enables using a standard off the shelf ICMP implementation and not have to modify it.
Our implementation has userspace ICMP request/response handling for echo request/response, it wasn't too hard to write. I don't propose modifying anything, but the structures were ~50 lines of code to set up. I agree we shouldn't be modifying responses we did not generate.
I think the existing text already is generic to "endpoint", not client or server: So this implies that both sides need to send the PING if they haven't guaranteed the MTU. This seems like this is already handled?
If both endpoints are required to send ping it would be resolved. However, that is not what is written in the draft currently. What is written is the there are various techniques, and if an endpoint choses something else than verifying its forward path using ping, it becomes unclear who is actually responsible for any HTTP intermediaries. Which results in an unclear requirement and very likely failure of the MTU verification resulting in an MTU black hole.
Well, it does say that endpoints MUST use this method unless the already have other coordination. I could see some clarification text to say that the guarantee for the other methods must include knowing there isn't an intermediary — which is actually pretty easy to guarantee in many cases.
That is what I intended when writing , but perhaps I wasn't clear. Can one of you propose a PR that would clarify this?
As per , the IPv6 minimum link MTU is 1280 bytes. However, states that it is possible to have a functional QUIC connection on a path that only allows 1200 bytes of UDP payloads. Such a path would only allow HTTP Datagrams with payloads up to 1180 bytes. If we only send IPv6 packets over QUIC DATAGRAM frames, such a setup would result in violating the IPv6 minimum link MTU. We should describe this problem and how to react to it. Possible solutions include failing the CONNECT-IP request, using the DATAGRAM capsule, or different solutions altogether.
This is only a problem for the end-2-end QUIC handshake packets, right? I was assuming that you either have a QUIC tunnel that has a larger MTU in order to use datagrams or you have to use stream-based transmission for the e2e handshake and then need a way to switch to datagram later. In any case we need to say more in the draft!
From the most recent session, I think we may have consensus to address this by noting that CONNECT-IP over H3 requires a minimum link MTU of X bytes on every HTTP hop for IPv6 compliance, leaving support for smaller links to an extension.
I'm not sure the consensus was clear, but I think it's upon us to make a proposal and see if it flies =)
That sounds like the best path forward.
I propose a simple solution that should be enough for most use-cases: if any endpoint detects that the path MTU is too low, it MUST close the tunnel if the tunnel had IPv6 addresses or routes on it. Endpoints are not required to measure the path MTU. endpoints that wish to use IPv6 SHOULD ensure that their path MTU is sufficient. For example, that can be guaranteed by padding QUIC initial packets to 1320 bytes.
That seems reasonable to me as a proposal!
So how does the endpoint determine that the Path MTU is too low? In the scenario that looks like this C - I1 -I2 - S and it is the I1-I2 path that can't support 1280 byte datagrams. The I nodes are HTTP intermediaries. The client will not know that the datagram MTU size is too low on these nodes, to my understanding the datagram payloads will simply be dropped by the I nodes in either direction.
In that scenario, if the endpoint wants to check it can send an ICMPv6 echo request with 1232 bytes of data - if it receives an ICMPv6 echo reply that means that the path supports the minimum IPv6 MTU.
NAME then we are making it mandatory to send ICMPv6 echo to the external address assigned for the client, and for the server sending one to the client, this to determine that minimal 1280 bytes are working over MASQUE. Because to me there are no way the MASQUE endpoint can determine in all cases if there are HTTP intermediaries or not. Else we have to put requirements on the HTTP intermediaries to understand MASQUE, and to my understanding what was not desired. And maybe using ICMPv6 is the easiest method with the lowest implementation burden. The issue I see with this aspect is that many applications when establishing a Connect-IP path might not want to wait for the ICMPv6 response before sending other traffic so we have a situation where there is a small, but non-zero risk that these packets goes into a black hole, and then the ICMP response comes back and the MASQUE Connect-IP stream will be closed due to the response to indicate the failure. I think this is all manageable and not different from failures that can occur for other reasons.
I don't see why this should be mandatory. When I plug in an Ethernet cable into my computer, I don't know whether the IP stack on the next router spuriously drops packets above 1000 bytes long, and my OS has no requirement to measure the path MTU.
But for the said router dropping IPv6 packets below 1280 would be a protocol violation. The issue here is the interaction with the HTTP intermediaries and what is under the control of Connect-IP function. So Connect-IP basically provides a IPv6 capable link. Per IPv6 RFC that requires us to support 1280 at least. What have been the rough consensus so far is that we are not defining an fragmentation and reassembly mechanism, or rather how to ensure that one uses QUIC streams when the transport is HTTP/3 for below 1280 IPv6 packets and instead reject the service. Thus we are in the position of having to answer how does one know when to reject the service. The alternatives I see are: 1) Force the Connect-IP endpoints to ICMP ping each other to verify that the path supports 1280 IPv6 packets 2) Have the HTTP intermediaries be Connect-IP aware to the level that they can reject the HTTP request if the path does not support 1280 packets. Where 2) have the downside that we still likely have some black holes in cases where the HTTP intermediaries are not connect-IP aware and ignores the requirements. Where 1) have the downside that the endpoints request can't be rejected during the request response sequence and instead have to be closed with an error message indicating the paths failure to support the packet.
I'm opposed to requiring ICMP pings because the majority of use-cases will not have this problem. How about we add text like this: CONNECT-IP endpoints that enable IPv6 connectivity in the tunnel MUST ensure that the tunnel MTU is at least 1280 bytes to meet the IPv6 minimum required MTU. This can be accomplished multiple different ways, including: Endpoints padding QUIC initial packets when it is known that there are no intermediaries present Endpoints sending ICMPv6 echo requests with 1232 bytes of data an out of band agreement between one of the endpoints and any involved intermediaries to have them pad QUIC initial packets
So I understand your desire to not add complexity anywhere for a likely seldom arising case. However, I do fear that not addressing this with a clear assignment on whose responsibility it is to avoid the issue will result in that none does anything of it. Because an endpoint that uses HTTP/3 can verify its first hop. However, the complexities arise when you have HTTP intermediaries in the mix.
I get that, but since the complexity is added by intermediaries we just need to make it clear that the intermediary needs to not be silly
One part is to detect if the link is capable of 1232 bytes, however, if not, we also not to provide more guidance what to do. I think there are two options: either close the connection or use quic streams instead of data datagrams.


If one is using IPPROTO to select a particular protocol to be tunneled, one may still received ICMP in the context of this request. To my understanding there are two reasons for this. a) ICMP generated by the Connect-IP peer endpoint b) ICMP sent back by node beyond the Connect-IP that is directly related to forward traffic sent by this endpoint and which the tunnel endpoint maps back to the Connect-IP request. I think we could add a clarification of this in , or we could do it later.
Totally agree. I added fb1578626de4d3073ac31f5cb89c06efae2ebc51 to address this as part of .
Based on the discussion in issue we recommend to use ICMP for error signal. ICMP works for MTU issue or destination unreachable, however, there is no appropriate message type if the IP header of the packet provided by the client contains an IP that wasn't assigned to the client. So we need a different kind of error handling for this case?
I think ICMP works for this, I'd send See . Would that work?
I guess that could do it. Should we be more specific in the draft? Also, we probably would need to be more explicit in the draft that you should request a scope for your CONNECT request that allows you to received ICMP messages.
Can you write a PR for this please?
IETF 114: Recommend including ICMP if providing routes to the peer, and clients should always be prepared to receive ICMP. Also add note that ICMP may have other source/destination addresses so as to not surprise endpoints.
Also look at RFC 4301 for some potentially helpful text.
Where is the ICMPv4 detail?


People shout at me for doing this, so I'm paying it forward. Perhaps you could move the SHOULD requirement to a statement in Section 3, so that it covers all forms of the CONNECT-UDP request.


In section 5 in this sentence the term "proxied UDP packets" is used the only time and might not be clear: How about to just say "UDP packets in HTTP datagrams"?
+1 good improvement


Thx!
Should we maybe add a recommendation to the sec considerations that proxies may limited the amount of data they send as long as they don't see any reply. I guess initially you would limit to 10 packets and then following RFC8085 limit to one datagram per 3 seconds...? Not sure if that really is a good idea but I wanted to bring it up for discussion...
I thought about that when I first wrote the existing text, and I came to the conclusion that a limit here wouldn't protect against real attacks. If someone is attacking a closed UDP port, they're not accomplishing anything. If someone attacks an open UDP port (for example 443 because they want to attack someone's QUIC endpoint), then the target will reply with some sort of UDP packet (for example a QUIC RETRY) so this protection will disable itself. Does that make sense?
The challenge I see with any advice is that the proxy is not an application using UDP, it's just passing opaque data. So it will really struggle to rationalise at all about the various UDP use cases that clients have for the proxy. Perhaps rather than focus on advice, focus on how CONNECT UDP proxy considerations differ from CONNECT TCP proxies. That may well cover, at the same time, considerations like how long should a proxy hold open requests (tunnels) that don't exchange any packets in one or both directions i.e., a DoS attack on the proxy itself via a slow loris kind of mechanism
That's discussed in the Proxy Handling section, it references BEHAVE for how long sockets should be kept open
NAME would it be worth adding your reasoning (which makes sense to me) to the draft somewhere?
fair point, I'd read that and forgotten about it, maybe worth a back reference from the security considerations. For instance, HTTP/2 has some special callouts to CONNECT in its security considerations URL edit: I'm not claiming the same considerations apply here, just that server resource usage as a byproduct of proxying might be different than other request loads
That makes sense, I've written up to address these comments.
Thank you


I also want to point an issue due to this stance. The fact that we are not explicit and discuss the expectations and realities of different IP versions we have interesting failure cases due to this text from Section 3.1: “However, if the target_host is a DNS name, the UDP proxy MUST perform DNS resolution before replying to the HTTP request.” No where in the follow paragraphs in this section do it discuss the slight issue that a DNS resolution can return only an AAAA record and the MASQUE server may only support IPv4. What is the error response? Should this be a proxy-status response. I would not there are currently no response for “IP version not supported”.
These concerns apply equally to TCP proxying via HTTP. We should strive to provide parity to how CONNECT treats these matters. HTTP semantics don't say anything
+1 to Lucas' meta-comment about striving for parity with CONNECT There is no such thing as a "unsupported IP version" DNS failure. Let's take an example where the client wants to CONNECT-UDP via the UDP proxy to the target . This particular host is configured such that querying AAAA will get a valid address, and querying A will yield a "no answer no error" DNS result. Let's assume that in this example the UDP proxy is IPv4-only. In that scenario, the UDP proxy will only query A (because it doesn't know what AAAA is) and it will get "no result no answer" so it will send
NAME in light of the previous comments, can you clarify what you think is missing from the document please?
First of all, I think that just answering lets do like Connect is a poor answer. There are benefits to alignment, but also one should ask if one are preventing useful functionality. In this case I think there are a relevant difference between what could be answered depending on how one implement stuff. Your suggest road says that for a IPv4 only proxy with a DNS name that has AAAA record but no A record the answer is: a) There is no answer For a node that would query both the answer could actually be: b) There is host, but I can't reach it as I don't support the right IP version, try another proxy. Sure, limiting ourselves to a) is easy, but is it the most useful for the protocol?
Disagree it's a poor answer. An HTTP proxy that inplements CONNECT for UDP is likely going to implement CONNECT for TCP. How to treat name resolution for both of these has to be consistent. The Proxy-Status header field is the appropriate way to signal such failures. Any text more complicated than what we have now, should be pursued in a separate document IMO. Not added at this late stage. It would be straightforward to form up a document like "DNS considerations for HTTP proxying" that is additive to the specifications we have.
I agree: Proxy-Status is the right place for this discussion, and CONNECT-UDP already recommends Proxy-Status. Also, as a technical matter, this is not how dual-stack connectivity works. As noted above, an IPv4-only UDP client will never issue a AAAA query for a destination hostname.
I agree with what Lucas and Ben said above. Let's move this discussion to an extension draft.
I agree that this discussion probably deserves to be in an extension draft. As is stands today, a DNS failure as described in this thread can just as easily result in a destination-unreachable / connection-closed error for the CONNECT-UDP stream, and an extension to provide detailed information as to why seems like the natural path forward.
Proxy status is absolutely the right thing to do here. The proxy-status is what we are already doing in production for CONNECT and CONNECT-UDP. The current text seems entirely appropriate for this: The only thing I could see saying further is to mention the error parameter, like this: I do not think we need any extension draft for this. This is very straightforward, and doesn't need to say much.
Thanks Tommy, I think that's a useful clarification. I wrote up PR based on your proposal to make this clearer
Ok, I am fine with this resolution.
Thank you for confirming Magnus!


This was prompted by .
LGTM. Nice and tight. URL Erik Kline has entered the following ballot position for draft-ietf-masque-connect-udp-14: No Objection When responding, please keep the subject line intact and reply to all email addresses included in the To and CC lines. (Feel free to cut this introductory paragraph, however.) Please refer to URL for more information about how to handle DISCUSS and COMMENT positions. The document, along with other ballot positions, can be found here: URL COMMENT: CC NAME Should the 2xx discussion here include additional reference to the extended commentary in masque-h3-datagram#section-3.2? I really wish there could be more said about tunnel security. I was tempted to try to formulate some DISCUSS on the matter, but I cannot seem to see where any more extensive text was ever written for the CONNECT method. Nevertheless, there are plenty of concerns that could be mentioned, like what should a proxy do if the target_host is one if its own IP addresses, or a link-local address, and so on. I'm not entirely convinced that it suffices to claim that because the tunnel is not an IP-in-IP tunnel certain considerations don't apply. Anything and everything can be tunneled over UDP, especially if RFC 8086 GRE-in-UDP is what the payloads carry (or ESP, of course). It seems like most of the points raised in RFC 6169 apply; perhaps worth a mention. But as I said, I think all these concerns apply equally to CONNECT (in principle, anyway), and the text here is commensurate with the text for its TCP cousin (that I could find, anyway).


Should the upgrade token be "connect-udp" or "masque-udp" ? (issue based on by NAME
I vote for or something along those lines. "MASQUE" is good as a working group name, but I don't think it's helpful in the protocol to help people understand what the variant of CONNECT does.


The default template's purpose is to enable compatibility with origin-only proxy configuration schemes. This is a worthwhile goal, but the default template is not a suitable solution: It violates RFC 8615's definition of .well-known/ as the location for metadata about an origin. A MASQUE service is not metadata. It is not representable as a .well-known "URI suffix" as required by the .well-known registry (unless we register all (2^128 + 2^32)2^16 distinct paths). It conflicts with better discovery mechanisms, like the one in the MASQUE development process. It is not compatible with the use of HTTPS records. (In the targeted legacy context, the client has no other DNS server that can be used when a proxy is in use.) This is somewhat ironic, as HTTPS records allow bootstrapping QUIC connections, and proxying for QUIC is the primary motivation for CONNECT-UDP. We should defer this problem to a subsequent draft, and provide a cohesive proxy setup flow, presumably with a standardized configuration object that could be served from .well-known/ and extended to cover CONNECT-IP, DoH and other relevant capabilities.
Fundamentally I disagree with your assessment that this solution isn't good and should be removed. This serves a real purpose. Let's go into the issues individually: Those points are not an argument to remove the default template, they are an argument to revert PR and switch back to a non-well-known prefix. Let's set those aside for now. I don't see a conflict here: the default template doesn't preclude using other templates that have been discovered through other means. Can you clarify what the conflict is? Can you clarify what the incompatibility is here? I don't understand what the relationship between the default template and HTTPS records is. Clients are welcome to query their own HTTPS records before using CONNECT or CONNECT-UDP. The way that the client picks its template has no impact on its DNS settings.
... This is called "squatting on the HTTP namespace", and it would violate the in RFC 7320. It's also messy to implement, as it would require mixing MASQUE-specific "routing" rules with the user's application-specific routing rules. It doesn't prevent better discovery mechanisms, but it doesn't encourage them either. From the server side, there would be no way to "opt out" of the default path, due to the potential for clients that have implemented this specification but not a later, better discovery mechanism. Servers that would prefer a different path would still have to operate the default path for backwards-compatibility. The relationship is that the default template cannot express the existence of a DNS server affiliated with the MASQUE server. No, they are not, as this would violate the privacy guarantees implicit in the legacy proxy settings UI that this feature is trying to preserve.
You're conflating two types of proxies here: traditional enterprise proxies that speak CONNECT today and can be updated to speak CONNECT-UDP privacy proxies designed to protect a user's privacy from the network Folks won't ever be deploying both of these use cases on the same proxy. Your concerns are valid but they only apply to (2). (2) would never use the default template. The default template only applies to (1).
Client software generally can't tell why a proxy is being used. It just knows that an HTTP CONNECT proxy has been configured with a particular protocol, domain, and port. Its behavior has to be correct for any existing use case. I don't see why privacy proxies wouldn't use the default template. Privacy proxies that are configured via the existing Secure Web Proxy configuration mechanisms (e.g. browser extensions, system settings) would have no other way to use MASQUE. Also, I think this distinction is false. Enterprises are free to use HTTP CONNECT proxies to conceal certain information from the network (e.g. using an employees-only browser extension), and enterprises are often opinionated about which DNS server ought to be used (not controllable via a browser extension).
For completeness, here's how I think this ought to look: URL
Your discovery draft sounds great - however the default template doesn't get in the way of that. I don't think it's reasonable to remove a useful feature because we might have a more complete solution later when the useful feature doesn't get in the way of the future complete solution.
Should the default URI template have a /.well-known prefix? For example, ? (issue based on by NAME
While you are at it..., how about something simpler? would put the host and port in query parameters AND it would have smaller, shorter names.
Having implemented this, placing the variables in the path is easier to parse on the proxy than placing them in query parameters
From the discussion in the room at IETF 113, it appeared folks were OK with . I'm going to pick that to close this issue, and we can open another issue during WGLC if we really want to bikeshed this further
It sound like you're writing your own URL parser? Any reasonable URL library will handle this for you.
I don't think placing these things in path components by default is reasonable.


In Section 2: When running over HTTP/1, requests are strictly serialized in the connection, therefore the first layer of demultiplexing is not needed. Is there any actual possibility to run more than a single request with HTTP datagram support over an transport connection? When one runs an HTTP request that enables datagram I don't see how it would be possible to close that request in any other way than closing the underlying transport connection. Or are there a way that I am missing that would enable one to close this request, and then issue another pipelined HTTP request that also uses HTTP datagrams? Is there a need to clarify this, or make it clear that only a single HTTP request with datagram can be sent?
In section 2, we've yet to mention anything about how the rest of the spec operates. So the comment in isolation is correct when talking about the general concept of multiplexin. I see your point but I'm not sure saying anything more here will help. Section 4.1 provides the details that implementers need to do the right thing.
Okay, but lets include Section 4.1 also into this. I assume you are referring to this sentence: In HTTP/1.x, the data stream consists of all bytes on the connection that follow the blank line that concludes either the request header section, or the 2xx (Successful) response header section. But, per URL this body length length rules will be those that could govern a closure of a HTTP request enabling datagram. Would it be realistic to actually make it clear that if one enable HTTP Datagram transport close is the only, way or is that just creating additional rules making implementation even more? Otherwise do we need additional analysis if there are possibility that any of these rules could actually result in a closing a HTTP request?
There is no HTTP content on upgrade requests, we should link to the definition of Upgrade URL and make things slightly more clear in Section 4.1.
Ok, I guess it becomes a question of which method for enabling datagram one uses, if this is an upgrade or a new method. And I would note that Connect method appears to have some interesting exceptions compared to other methods. Which I think makes this into a requirement on the using method/upgrade that uses HTTP datagrams to actually specify this aspect.
I've written up to clarify that pipelining is not supported.


When reviewing the latest editors copy, the document structure leaps out at a bit odd by jumping from section 3 to section 5. That indicates to me these concepts are linked and also independent of the capsule protocol. So here I propose a minor restructure by moving the H3_DATAGRAM setting stuff to be a subsection of the HTTP/3 datagram format. This makes all the top-layer sections loosely coupled, which is nicer in my opinion.


The draft currently states: However in PR , NAME points out that this "state" is not well defined. I agree, we should fix it. Thinking about this more, I don't think we need the concept of related state. We can instead just say that received datagrams MUST be silently dropped if the stream receive side is closed.
I agree with removing the concept of state is better. I don't want to be overly nitty and the proposed sentence is better but there also nothing like "the stream receive side", as there can be multiple streams (open and closed), that's why I proposed Or if we don't want state, then it would be:
That's not quite right: there cannot be multiple streams, as datagrams are strongly associated with a client-initiated bidirectional stream. I'd phrase the paragraph as: open. If a datagram is received after the receive side of the corresponding stream is closed, the received datagrams MUST be silently dropped.
That works. Thanks!
This answer is necessary because QUIC doesn't guarantee ordering, but is this necessarily true indefinitely? But can I regard a datagram that arrives an hour after the stream closes as an error?
If you keep track of QUIC packet numbers, you can see that the peer closed the stream before it sent the datagram which means they violated a MUST. You're welcome to error if you detect that, but I don't think we should mandate this level of tracking.
I'm not sure that that is a good strategy given how some deployments manage their packet numbers. I'm talking about maybe running a really long timer.
That introduces a way for an attacker to cause connection closure by delaying a specific packet. What problem are you trying to solve?
Good point. I thought that we did similar in a few places in QUIC. As you say, it's possible to enforce, but tricky. The concern I'd have is that this effectively forbids enforcement, but maybe that's OK.


End of section 3.5 () says: First of all this paragraph seems to be in the wrong section because it's more generally about the capsule protocol and not the datagram capsule. However, it's also confusing. I believe if you negotiate http datagram with the H3DATAGRAM SETTINGS parameter, you always need to also support capsules even if QUIC datagrams are supported, as the other end (e.g. an intermediary) could send you anyway a capsule. Or is this text meant to say that new extensions that use the http datagram but have a new settings parameter for negotiation may not need to support the capsule protocol? I think it would be good to clarify both: what this sentence/paragraph means and what is required if you negotiate http datagrams using the H3DATAGRAM SETTINGS parameter.
I want to extended on this issue a bit more. A lot about the implementation requirement on different nodes are implied rather than being explicit. "Secondly, I will note that the above text combined with what is written in Section 3.4: Intermediaries MAY use this header field to allow processing of HTTP Datagrams for unknown HTTP Upgrade Tokens; note that this is only possible for HTTP Upgrade or Extended CONNECT." If an intermediary is doing this unaware re-encoding and the HTTP extension is an extended connect version it can result in interoperability failure when a Datagram capsule arrives at the endpoint that only supports HTTP/3 Datagrams over QUIC Datagram.
If we build off the notion that H3DATAGRAM setting communicates a willingness to receive the HTTP/3 datagrams (in QUIC datagrams), they still require some request to bootstrap the association, otherwise those datagrams would be ignored. In theory, I could define a new method called "ALWAYSUNRELIABLE" that can be used to bootstrap a request stream and it be defined to never use a capsule protocol. If a server sends me data on that stream, I could ignore it or throw an error. I don't need a new setting to caveat this behaviour.
NAME but then you can't have "blind" re-encoding of HTTP/3 Datagram to Datagram capsules, I think it is that simple. So we need to choose if HTTP Intermediaries can re-encode when they don't know the Upgrade Token or if they can't and must be aware and thus enable cases where some have different behaviors. See
There is no such thing as "blind re-encoding". Intermediaries can only perform re-encoding when they know that the capsule protocol is in use. Intermediaries can learn that by knowing the HTTP Upgrade Token, or by watching for the Capsule-Protocol header. But I think we can make that clearer by spelling out in the "DATAGRAM Capsule" section that intermediaries MUST NOT re-encode to capsules unless they know for sure that the capsule protocol is in use.
PR did not address the editorial point that this paragraph is in the wrong section.
I don't agree with that editorial point. The paragraph is about the interaction between HTTP datagrams and the capsule protocol, so having in the DATAGRAM capsule section makes sense to me.
It about the interaction about http datagrams (section 2) and the capsule protocol (section 3). There is nothing related to datagram capsules (section 3.5) in this paragraph. I guess you could just add a new subsection 3.6 if you prefer to have it at the end.
LGTM


Written in response to
From in reference to the text in this PR: Issues ietf-wg-masque/draft-ietf-masque-connect-udp-listen (+0/-2/0) 2 issues closed: Improve document title URL [editorial] Should "connect-udp-listen" be registered as "Connect-UDP-Listen"? URL ietf-wg-masque/draft-ietf-masque-quic-proxy (+0/-0/2) 1 issues received 2 new comments: Virtual connection IDs in multi-hop settings (2 by DavidSchinazi, mirjak) URL Pull requests ietf-wg-masque/draft-ietf-masque-connect-udp-listen (+2/-2/1) 2 pull requests submitted: Update Listener to bind (by asingh-g) URL update connect-udp-listen to use H1 style (by asingh-g) URL 1 pull requests received 1 new comments: Compress away IP and Port using Context IDs (1 by asingh-g) URL 2 pull requests merged: Update Listener to bind ( URL update connect-udp-listen to use H1 style, URL ietf-wg-masque/draft-ietf-masque-quic-proxy (+0/-0/3) 1 pull requests received 3 new comments: Proxy-Chosen Virtual Client Connection ID (3 by DavidSchinazi, ehaydenr) URL Repositories tracked by this digest: URL URL URL URL URL URL URL
Roman Danyliw has entered the following ballot position for draft-ietf-masque-h3-datagram-10: No Objection When responding, please keep the subject line intact and reply to all email addresses included in the To and CC lines. (Feel free to cut this introductory paragraph, however.) Please refer to URL for more information about how to handle DISCUSS and COMMENT positions. The document, along with other ballot positions, can be found here: URL COMMENT: Thank you to David Mandelberg for the SECDIR review. ** Section 4. Consider adding the following clarification: NEW HTTP Datagrams and the Capsule Protocol are building blocks for HTTP extensions to define new behaviors or features and do not constitute an independent protocol. Any extension adopting them will need to perform an appropriate security analysis which considers the impact of these features in the context of a complete protocol.


This PR aims to take the design discussions we had at the last interim, and propose a potential solution. This PR should in theory contain enough information to allow folks to implement this new design. That will allow us to perform interop testing and confirm whether the design is sound.
I'm confused by this pull request (probably because I was out of the loop for a while). My understanding, at least when I first read the proposal a month ago was that we would separate multiplexing streams and multiplexing contexts into different layers, so that methods that need context would get it (CONNECT-UDP), and features that don't (WebTransport/extended CONNECT) could use stream-associated datagrams directly.
NAME you're referring to which this PR isn't trying to address. That'll happen in a subsequent PR.
In , capsules, contexts and context IDs are explicitly defined as end-to-end and there are requirements for intermediaries to not modify them, and to not even parse them. The motivation for this was to ensure that we retain the ability to deploy extensions end-to-end without having to modify intermediaries. NAME mentioned that these requirements might be too strict, as he would like to have his intermediary be able to parse the context ID, for logging purposes.
Part of the comment from the PR is also about consistency of interface for generic HTTP libraries that can be used to build either endpoints or intermediaries. If the context ID is always present, it's overly prescriptive to say that an intermediary can't parse it, and such a requirement couldn't be enforced anyways. If context IDs are negotiated on/off, whether this is possible depends on the negotiation mechanism.
I think its fine for a component that has bytes passing through it to look at those bytes. The import thing to do is set expectations on what that means for the protocol behaviour we are defining. We don't expect intermediaries to do anything with a context ID; they should not enforce the protocol semantic rules (odd, even, reuse etc). H2 and H3 somewhat solve this problem by decoupling frame parsing from frame payload interpretation. E.g. if the form of a request or response is invalid, an intermediary shouldn't pass it on. DATAGRAM is different because it isn't defined as an actual HTTP/3 frame with frame headers and frame payload. Something that might work is to formalise that quarter stream ID is part of the envelope that intermediaries have to be able to handle the syntax of, but context ID is part of content and separate.
Unless you encrypt capsules end-to-end, I don't think any text in the spec can in practice prevent intermediaries from inspecting and modifying capsules. You can send something like GREASE capsules, though.
In , we introduce the CLOSEDATAGRAMCONTEXT capsule which allows endpoints to close a context. That message currently only contains the context ID to close. NAME suggests that we may want to add more information there. Should we differentiate between "context close" and "context registration rejected"? Should we add an error code? In particular, do folks have use cases for these features?
My first concern is with infinite loops. If the context was "garbage collected" by the recipient, the sender can simply re-create it if they still want it. However, if it was closed due to incompatibility (i.e. rejected) or hitting a resource limit (e.g. max # of contexts), reopening it in response will produce a tight infinite loop. This could be maybe be avoided by some kind of heuristic, but an explicit indication of what's going on seems better.
This issue assumes that decide to go with the two-layer design described in . Given that design, some applications might not need the multiplexing provided by context IDs. We have multiple options here: Make context ID mandatory Applications that don't need it waste a byte per datagram Negotiate the presence of context IDs using an HTTP header Context IDs would still be mandatory to implement on servers because the client might send that header Have the method determine whether context IDs are present or not This would prevent extensibility on those methods Create a REGISTERNOCONTEXT message This assumes we use the Message design from We add a register message that means that this stream does not use context IDs This allows avoiding the byte overhead without sacrificing extensibility
Making the context ID mandatory is simple, although slightly wasteful. For the variations where it is optional, I prefer the one that uses a header. It could work with a variant of where we do Header + Message, and the Header determines whether or not you have contexts, and whether or not they can be dynamic.
What are the benefits of message over header to make it optional? To me it seems like header is sufficient, because the context IDs are end-to-end, and should not affect intermediaries, making the header a natural place to put it. I think having the context ID be the default behavior and the header is to opt-out is the right decision if we were to allow removing it.
This is a bit of a bike-shed, but why would it default to having context-IDs instead of not having them?
I think the second option "Negotiate the presence of context IDs using an HTTP header" is the best personally. In terms of: "Context IDs would still be mandatory to implement on servers because the client might send that header", I assume server in this context is the final application server, not a proxy? Even so, I'd argue it's applications which decide if they need this functionality, so if a server only implemented one application and it didn't use Context-IDs, then I don't see why it would need to implement it.
Ian, in your comment, are you talking about an intermediary when you say "proxy"? If so, then an intermediary need not implement anything based on the payload of than H3 DATAGRAM, it only has to pass the data along. That may be worth a clarification.
Having context ID optional has a nice future-proof property - other uses of datagrams in HTTP/3 can decide if they want the second layer or not. I wouldn't even state it as application needing multiplexing or not. Other applications might find it very useful to include a second ((third, fourth, fifth) layer of frame header. I think its fine for us to state as much in this in the spec, while also defining an explicit second layer for multiplexing.
Yes, by proxy I meant intermediary. And I agree about what intermediaries need to do, I just wanted to ensure I understood what server meant when David used it above. Yup, whenever all this gets written up, we can make it clear that intermediaries can ignore much of this.
The method defines the semantics of datagrams for a particular request, thus I think the third option makes most sense. I don't see a point in using explicit negotiation, since in the two cases I can think of the top of the head, the answer is always clear from the method (CONNECT-IP seems to always want context IDs, WebTransport does not ever want context IDs). If some method starts out without context, but wants to add context later, it can add its own negotiation.
One set of use cases for which the method approach(3) is best are those where there are multiple context IDs but they're predefined by the method and don't need negotiation. My strawman is WebRTC.
During the 2021-04 MASQUE Interim, we discussed multiple options for registering context IDs. Without going into the specific of the encoding, there are two broad classes of solutions: "Header" design Once-at-setup registration This happens during the HTTP request/response Simplest solution is to use an HTTP header such as (encoding TBD) Example use-case: CONNECT-UDP without extensions "Message" design Mid-stream registration This can happen at any point during the lifetime of the request stream This would involve some sort of "register" message (encoding TBD, see separate issue) Example use-case: CONNECT-IP on-the-fly compression (this can't be done at tunnel setup time because the bits top be compressed aren't always known at that time) It's possible to implement once-at-setup registration using the Message design, but it isn't possible to implement mid-stream registration. Therefore I think we should go with the Message design. It would also be possible to implement both but I don't think that provides much value.
I'd like to here some more from the WebTransport use case on the issue. My understanding is that the Requests are used to create WebTransport sessions and that creation of datagram flows(contexts) could be a lot more ad-hoc / server generated after the request phase, when compared to the CONNECT-UDP case. Cc NAME
Personally, I'd vote for either "message" or "message + header" (a la priorities), but not just "header". Mid-stream annotation of context/flow/etc is quite useful.
I prefer "Message" only, since the dynamic use case seems like a fairly compelling use of sub-flows(contexts?). I'm not sure there are many cases when one allocates all sub-flows up front AND needs to use a header to do that in a flexible way. If an application needs multiple sub-flows, but they're fixed in number/format(ie: data and control), I don't think there's a need for a header or a "Message", since the application can consider that part of its internal format.
During the 2021-04 MASQUE Interim, we discussed having the ability to accept or reject registration of context IDs. An important motivation for this is the fact that context ID registrations take up memory (for example it could be a compression context which contains the data that is elided) and therefore we need to prevent endpoints from arbitrarily making their peer allocate memory. While we could flow control context ID registration, a much simpler solution would be to have a way for endpoints to close/shutdown a registration.
IIUC there's an opposing view that the constraints can be negotiated apiori. Either statically (such as defining an extension which states that there can only be N contenxts of a given type) or dynamically using something like a HTTP header. Failure to meet these constraints could lead to an HTTP/3 stream or connection error rather than a flow rejection.
NAME I've filed to discuss whether to register/negotiate at request start or mid-stream.
I think that if there is the ability to register mid-stream (), we need the ability to close/de-register/reject mid-stream as well. If we don't have mid-stream changes, then no need to negotiate. However, someone trying to register something needs to be able to know whether or not the thing they registered will be accepted.
I agree that notification of the registration status is particularly important if we allow mid-stream registration.
In draft-ietf-masque-h3-datagram-00, flow IDs are bidirectional. During the 2021-04 MASQUE Interim, we discussed the possibility of making them unidirectional. Here are the differences: Bidirectional single shared namespace for both endpoints (even for client, odd for server) it's easier to refer to peer's IDs, which makes negotiation easier Unidirectional one namespace per direction every bidirectional use needs to associate both directions somehow I'm personally leaning towards keeping them bidirectional. Unlike QUIC streams, these do not have a state machine or flow control credits, so there is no cost to only using a single direction of a bidirectional flow for unidirectional use-cases.
There's another option where flows are unidirectional but they share a namespace (cleint even/server odd). I'm undecided. And I also don't know how important it is to choose. The 2 layer design means that DATAGRAMs always have a bidirectional tie via the stream ID. On balance, I think bidirectional would be good enough today but I would be receptive to extension use cases that make a strong point of unidirectional.
Bidirectional is my personal preference, but either way could work ultimately. It seems to me that any extension that wants unidirectional can just only use this context/flow/etc ID for one direction if it wants.
I think bidirectional flows are likely overcomplicated and unnecessary. For example, if either peer can "close" a flow, this requires a much more complex state machine. In my experience, half-closed states are hard to get right. Closing unidirectional flows seems much easier to reason about. Many, perhaps most, of our use cases for flows are really unidirectional. For example, DSCP marking likely only makes sense from client to server, while timestamps for congestion control tuning likely only make sense from server to client. QUIC connection IDs are different in each direction, so QUIC header compression may be easiest to understand with unidirectional flows.
Bidirectional doesn't imply half-close states. I'm thinking of a bidirectional flow that can only be closed atomically as a whole. I don't think that's true. The default extension-less CONNECT-UDP is bidirectional. So is ECN, and so would a timestamp extension.
I would prefer bidirectional without support for half-close. I feel that having unidirectional flow/context IDs adds overhead we don't need; peers now need to track those associations and it's unclear what the benefits of having them separated out are.
This issue is conflating consent (unilateral/bilateral) with scope (unidirectional/bidirectional). IIUC there are two questions here: 1) Can I send a flow-id forever once declared, or can the endpoint tell me to stop without killing the whole stream (consent)? 2) Can the receiver of a flow-id assignment use that same flow-id when sending datagrams, or could this mean something completely different? I think the answer to (1) is "yes, it can tell me to stop" (i.e. it is negotiated while allowing speculative sending). For (2), I lean toward bidirectional, but not strongly. There are plenty of codepoints, so it seems wasteful for each endpoint to define the same semantic explicitly.
NAME the intent of this issue wan't to conflate those two things. This issue is about scope. For consent, see issue .
Conceptually, I like unidirectional better. I tell the peer that when I send with context-ID=X, it has a given semantic. For bidirectional the sender is saying both that and that if the peer sends on that ID the original sender will interpret with the same semantic. That might not even make sense. Is it possible to associate datagrams with a unidirectional stream (push?). If so, does allowing for bidirectional context imply that even though the receiver cannot send frames on the stream, it can send datagrams? Every negotiation does run the risk of loss/reordering resulting in a drop or buffering event at the receiver. The appeal of bidirectional is that it removes half of this risk in the cases where the bidirectionality is used.
To second NAME I would also prefer bidirectional without support for half-close. Given you can always not send something in one direction, and there's none of the overhead of streams(ie: flow control, etc) I can't see any benefits of unidirectional over bidirectional in this case. Associating h3-datagrams with a server push seems like unnecessary complexity to me, but you're right that it should be clarified.
During the 2021-04 MASQUE Interim, we discussed a design proposal from NAME to replace the one-layer flow ID design with a two-layer design. This design replaces the flow ID with two numbers: a stream ID and a context ID. The contents of the QUIC DATAGRAM frame would now look like: While the flow ID in the draft was per-hop, the proposal has better separation: the stream ID is per-hop, it maps to an HTTP request the context ID is end-to-end, it maps to context information inside that request Intermediaries now only look at stream IDs and can be blissfully ignorant of context IDs. In the room at the 2021-04 MASQUE Interim, multiple participants spoke in support of this design and no one raised objections. This issue exists to ensure that we have consensus on this change - please speak up if you disagree.
I am fine with the 2-layer design for CONNECT-UDP, but I am not sure if the second identifier should be in H3 DATAGRAM or should be specific to CONNECT-UDP. As I indicated to the list, webtrans and httpbis might weigh in on this with broader consideration of other use cases.
Hi NAME the topic of the optionality of context IDs is discussed in issue .
I think this is a good design, because it minimizes the core functionality of the draft, but avoids every application that needs multiple dynamically created sub-flows from re-inventing a different mechanism. Nit: I would not call this a 'Frame', since it's not an HTTP(TLV) frame.
I think this is a good change. Thanks! There are some things that I think we should still discuss, but we shouldn't hold up this change on those details.


NAME points out that in order to correctly forward DATAGRAMs between transports where the QUIC DATAGRAM frame isn't always available, intermediaries need the ability to parse some capsules.
CAPSULE was easier to understand when it was fully opaque. When should I choose to define a semi-opaque CAPSULE vs defining a new frame? Aren't both constructs effectively hop-by-hop?
NAME you're the one who asked for this :-) The benefit of a semi-opaque capsule is that it allows defining new capsules without having to update every single intermediary in the chain, because intermediaries forward all unknown capsules.


Had offline conversation with Lucas, conclusion was that this is good enough to merge to allow interop - we'll have plenty of time to figure out details in subsequent issues and PRs.


LGTM
From NAME The very first sentence is mostly unparseable due to its length (7 lines). Please consider splitting it.
I added this, from NAME review: This section and the next one should be a list with 2 bullets rather than 2 sections to make the text more readable.


From NAME Please consider using the same wording for the title as for 6.2 and 6.1 (nit) Who is "we" in " We do have experience" ? While I share some concerns about deploying a new transport in user space as opposed to more stable and controlled kernel space, I wonder whether this paragraph is relevant in this document. Does it bring any value ? If no, then please remove it.
We'll look at the date for CUBIC being enabled by default in the Linux kernel as "experience". We'll delete the paragraph as suggested.
was fixed elsewhere.
LGTM. I'm almost sad to lose the ref to "Congestion Defense in Depth", but I guess it makes good sense, that's not a consensus position.


Fixed a few typos, otherwise ready to merge.
Merging on Ali's review.
From NAME (URL) (In Section 3.6. Unpredictable Usage Profiles / Section 3.7. Extremely) Unpredictable Usage Profiles I am not fully convinced by the distinction between "unpredictable" and "extremely unpredictable". To me, these two sections could be merged and maybe also be shortened. More specifically, Section 3.7 lists a lot of statistics from the past. What is IMHO a bit missing in Section 3.7 are actual operational considerations. For instance, are there any lessons learnt for the future? Note that Section 3.6 has a reasonable conclusion at the end. Reviewer: Michael Scharf Review result: Ready with Issues This document has been reviewed as part of the transport area review team's ongoing effort to review key IETF documents. These comments were written primarily for the transport area directors, but are copied to the document's authors and WG to allow them to address any issues raised and also to the IETF discussion list for information. When done at the time of IETF Last Call, the authors should consider this review as part of the last-call comments they receive. Please always CC EMAIL if you reply to or forward this review. This informational document surveys network and transport protocol issues that affect quality of experience for streaming applications, in particular video. The overview covers many topics across different technologies. Readers may only be familiar with a subset of these topics and could therefore learn quite a bit from this kind of tutorial. Nonetheless, it remains somewhat unclear what the actual objective of some of the text is. For some topics, quite a bit of technical background is provided, but the discussion is not really comprehensive. In many sections, the document neither derives technical challenges, nor system/protocol requirements, nor practical usage guidance. It is just a kind of tutorial. Quite a bit of text also presents ongoing IETF work, and an RFC with this scope may thus get outdated soon. Section 6.2 deals with topics owned by TCPM. In similar past documents, I have asked for a TCPM presentation prior to an IETF last call in order to ensure that owners of running code are in the loop. I believe this strategy has worked well in the past. Having said this, as TSV-ART reviewer I don't strongly disagree with a publication. All these issues may just be OK for an informational RFC. Some more specific comments: Section 3.2.1. Recognizing Changes from an Expected Baseline This section apparently assumes relatively static path properties, e.g., fixed network connectivity. It would be more useful to analyze the impact of Wifi and 4G/5G networks in one place. Some of this follows later in sections 3.2.1 and Yet, the split between these three sections is unclear. Having a discussion about today's Internet path characteristics and the dynamics at one place could be more useful. The text could also better distinguish between what matters to endpoints and considerations relevant only for middleboxes (such as passive monitors). Section 3.3 Path Requirements The statement "to find the bandwidth requirements for a router on the delivery path" may assume that the bottleneck on the path will be routers. Yet, on a path the bottlenecks could also be a link layer device (e.g., an Ethernet Switch, a Wifi access point, etc.). RFC 6077 (specifically Section 3.1.3) also explains that issue. A better wording may be "to find the bandwidth requirements for a delivery path". Section 3.6. Unpredictable Usage Profiles / Section 3.7. Extremely Unpredictable Usage Profiles I am not fully convinced by the distinction between "unpredictable" and "extremely unpredictable". To me, these two sections could be merged and maybe also be shortened. More specifically, Section 3.7 lists a lot of statistics from the past. What is IMHO a bit missing in Section 3.7 are actual operational considerations. For instance, are there any lessons learnt for the future? Note that Section 3.6 has a reasonable conclusion at the end. Section 4. Latency Considerations and Section 4.1. Ultra Low-Latency I am surprised by the definition "ultra low-latency (less than 1 second)", as well as some of the other numbers. For other real-time communication use cases, "ultra low-latency" would probably imply a latency requirement of the order of one millisecond. For instance, isochronous traffic for motion control in industrial networks may require a latency of 1 ms, and such "ultra-low latency" requirements are discussed elsewhere, e.g., in the DetNet WG. The terminology in this section should be better explained to readers dealing with networked systems with much harder latency requirements. And a reference should be added for these definitions in the context of video streaming. Section 5.5.2 Head-of-Line Blocking If Head-of-Line Blocking is indeed a relevant operational problem, it would be useful to add a corresponding reference (e.g., with measurements). Section 5.5.3. Wide and Rapid Variation in Path Capacity The statement "As many end devices have moved to wireless connectivity for the final hop (Wi-Fi, 5G, or LTE), new problems in bandwidth detction have emerged from radio interference and signal strength effects." could be moved to earlier parts of the document. Quite a bit of new computers apparently only have Wifi connectivity and no Ethernet port, i.e., Wifi may just be their default. Also, wireless may not only be used on the last hop, for instance in meshed setups. This may make the problem even harder. Section 6. Evolution of Transport Protocols and Transport Protocol Behaviors I really wonder whether UDP vs. TCP vs. QUIC is actually the relevant distinction. What may actually matter are the transport services provided by the protocol stack (e.g., RFC 8095). I fully agree to a related comment in the INTDIR review. Section 6.1. UDP and Its Behavior UDP is also used for encrypted tunnels (OpenVPN, Wireguard, etc.). Does encrypted tunneling really have no operational impact on streaming apps other than circuit breakers? (I am thinking about 4K streaming over OpenVPN and the like.) Section 6.2. TCP and Its Behavior This text should IMHO be presented and be discussed in TCPM. Personally, I am not convinced that this document is a good place for discussing the long history of TCP congestion control. If the objective of the document is to provide a tutorial, IMHO it would be more useful to briefly explain the state-of-the-art in year 2022. Some further specific comments on TCP congestion control: It is surprising that RFC 5681 is not referenced.793bis has normative statements regarding congestion control, and most stacks are compatible to what is written in 793bis. Several operating systems use CUBIC by default and support standard Reno. Another general good reference for TCP standards is RFC 7414.Does ECN, DCTCP, etc. really not matter at all, for instance, if a data center hosts video playout servers? Section 6.3. QUIC and Its Behavior As already noted, the discussion about head-of-line-blocking would really benefit from backing by a reference. Section 7. Streaming Encrypted Media As far as I know, streaming users sometimes use encrypted tunnels such as OpenVPN or WireGuard (or IPsec) to access video content. I may miss something, but it is unclear to me how that fits into the categories presented in Section


NAME - several things here. we probably don't need all of the descriptive text - the idea is that we'd say "assume roughly this protocol stack, and here's what's not included" if distinguishing between "transport protocol" and "media transport protocol" makes sense, that probably needs to be reflected elsewhere in the document (like, a lot of elsewheres) If this approach works, it could either be in the transport protocol section, or in the one you're thinking of for
I don't think there will be a lot of places. We need to pay attention to only the places where we are explicitly referring to the media transport protocol. The rest can stay the same IMO.
NAME - I couldn't reply to this inline, but I looked through the document (searching for "transport protocol", and you're right. I found one place in Section 4.1, and changed the description of RTP as a transport protocol to RTP as a media transport protocol. NAME - Section 4.1 is before where my explanation is now - if this PR makes sense to you, the explanation probably DOES belong much earlier in the doc, perhaps in .
I'm merging this, based on Ali's review.
We want a paragraph to say something about scope along the lines of: This document is for endpoints delivering streaming media traffic. This can include intermediaries like CDNs as well as content owners and end users, plus their device and player providers. We think this can help address a few issues from last call/iesg review, including: (more to be linked as we notice them)
Consider adding a subsection in the introduction about things that are out of scope, and (for extra credit), why they are out of scope.
from NAME 's review (URL) Section 6.1. UDP and Its Behavior UDP is also used for encrypted tunnels (OpenVPN, Wireguard, etc.). Does encrypted tunneling really have no operational impact on streaming apps other than circuit breakers? (I am thinking about 4K streaming over OpenVPN and the like.)
Not sure if this is really about scope clarification? I think this section is intending to talk about usage as part of a transport protocol, but tunnels are sort of more like a path overlay issue or something (might affect mtu, extra hops with extra things that can go wrong, etc.).
NAME - I'm looking more closely at this issue, and I think text that I put in another (individual) draft may be useful here as well. I'll add it in this section (making the obvious substitutions), and I'll see what NAME and NAME think (after the holiday weekend), and then check back with you. If it works for people, the result will likely be clearer to many readers. Reviewer: Michael Scharf Review result: Ready with Issues This document has been reviewed as part of the transport area review team's ongoing effort to review key IETF documents. These comments were written primarily for the transport area directors, but are copied to the document's authors and WG to allow them to address any issues raised and also to the IETF discussion list for information. When done at the time of IETF Last Call, the authors should consider this review as part of the last-call comments they receive. Please always CC EMAIL if you reply to or forward this review. This informational document surveys network and transport protocol issues that affect quality of experience for streaming applications, in particular video. The overview covers many topics across different technologies. Readers may only be familiar with a subset of these topics and could therefore learn quite a bit from this kind of tutorial. Nonetheless, it remains somewhat unclear what the actual objective of some of the text is. For some topics, quite a bit of technical background is provided, but the discussion is not really comprehensive. In many sections, the document neither derives technical challenges, nor system/protocol requirements, nor practical usage guidance. It is just a kind of tutorial. Quite a bit of text also presents ongoing IETF work, and an RFC with this scope may thus get outdated soon. Section 6.2 deals with topics owned by TCPM. In similar past documents, I have asked for a TCPM presentation prior to an IETF last call in order to ensure that owners of running code are in the loop. I believe this strategy has worked well in the past. Having said this, as TSV-ART reviewer I don't strongly disagree with a publication. All these issues may just be OK for an informational RFC. Some more specific comments: Section 3.2.1. Recognizing Changes from an Expected Baseline This section apparently assumes relatively static path properties, e.g., fixed network connectivity. It would be more useful to analyze the impact of Wifi and 4G/5G networks in one place. Some of this follows later in sections 3.2.1 and Yet, the split between these three sections is unclear. Having a discussion about today's Internet path characteristics and the dynamics at one place could be more useful. The text could also better distinguish between what matters to endpoints and considerations relevant only for middleboxes (such as passive monitors). Section 3.3 Path Requirements The statement "to find the bandwidth requirements for a router on the delivery path" may assume that the bottleneck on the path will be routers. Yet, on a path the bottlenecks could also be a link layer device (e.g., an Ethernet Switch, a Wifi access point, etc.). RFC 6077 (specifically Section 3.1.3) also explains that issue. A better wording may be "to find the bandwidth requirements for a delivery path". Section 3.6. Unpredictable Usage Profiles / Section 3.7. Extremely Unpredictable Usage Profiles I am not fully convinced by the distinction between "unpredictable" and "extremely unpredictable". To me, these two sections could be merged and maybe also be shortened. More specifically, Section 3.7 lists a lot of statistics from the past. What is IMHO a bit missing in Section 3.7 are actual operational considerations. For instance, are there any lessons learnt for the future? Note that Section 3.6 has a reasonable conclusion at the end. Section 4. Latency Considerations and Section 4.1. Ultra Low-Latency I am surprised by the definition "ultra low-latency (less than 1 second)", as well as some of the other numbers. For other real-time communication use cases, "ultra low-latency" would probably imply a latency requirement of the order of one millisecond. For instance, isochronous traffic for motion control in industrial networks may require a latency of 1 ms, and such "ultra-low latency" requirements are discussed elsewhere, e.g., in the DetNet WG. The terminology in this section should be better explained to readers dealing with networked systems with much harder latency requirements. And a reference should be added for these definitions in the context of video streaming. Section 5.5.2 Head-of-Line Blocking If Head-of-Line Blocking is indeed a relevant operational problem, it would be useful to add a corresponding reference (e.g., with measurements). Section 5.5.3. Wide and Rapid Variation in Path Capacity The statement "As many end devices have moved to wireless connectivity for the final hop (Wi-Fi, 5G, or LTE), new problems in bandwidth detction have emerged from radio interference and signal strength effects." could be moved to earlier parts of the document. Quite a bit of new computers apparently only have Wifi connectivity and no Ethernet port, i.e., Wifi may just be their default. Also, wireless may not only be used on the last hop, for instance in meshed setups. This may make the problem even harder. Section 6. Evolution of Transport Protocols and Transport Protocol Behaviors I really wonder whether UDP vs. TCP vs. QUIC is actually the relevant distinction. What may actually matter are the transport services provided by the protocol stack (e.g., RFC 8095). I fully agree to a related comment in the INTDIR review. Section 6.1. UDP and Its Behavior UDP is also used for encrypted tunnels (OpenVPN, Wireguard, etc.). Does encrypted tunneling really have no operational impact on streaming apps other than circuit breakers? (I am thinking about 4K streaming over OpenVPN and the like.) Section 6.2. TCP and Its Behavior This text should IMHO be presented and be discussed in TCPM. Personally, I am not convinced that this document is a good place for discussing the long history of TCP congestion control. If the objective of the document is to provide a tutorial, IMHO it would be more useful to briefly explain the state-of-the-art in year 2022. Some further specific comments on TCP congestion control: It is surprising that RFC 5681 is not referenced.793bis has normative statements regarding congestion control, and most stacks are compatible to what is written in 793bis. Several operating systems use CUBIC by default and support standard Reno. Another general good reference for TCP standards is RFC 7414.Does ECN, DCTCP, etc. really not matter at all, for instance, if a data center hosts video playout servers? Section 6.3. QUIC and Its Behavior As already noted, the discussion about head-of-line-blocking would really benefit from backing by a reference. Section 7. Streaming Encrypted Media As far as I know, streaming users sometimes use encrypted tunnels such as OpenVPN or WireGuard (or IPsec) to access video content. I may miss something, but it is unclear to me how that fits into the categories presented in Section


…transport protocols Also tighten up description of the assumed protocol stack in (what is now) Section 6.1. Note that the UDP and TCP sections have been flipped in order, to improve section flow and clarity.
NAME - If you have a moment before our call on Friday to take a look at this PR, I'd appreciate it. I think it's important to to get that right, in order to address both the INTDIR and TSVART reviews.
From NAME As a bit of transport commentary, I don't find the setup of Section 6 compelling. While UDP is a transport in a technical sense, for the purpose of media, it is almost always a layer upon which the protocol doing the congestion control work runs. To that end, QUIC is just another more standardized case of this, just like SCTP over UDP. Rather than talking about "UDP's behavior" vs "TCP's behavior", I suggest talking about how applications over UDP for media behave vs applications over TCP for media behave.
NAME - I need to look at this more closely. I understand your point about UDP, but (because UDP is not a real transport layer, while TCP and QUIC are), I suspect (without looking closely yet) I'm going to be making sure that I explain this more clearly, without trying to explain the behavior(s) of applications running over TCP and QUIC. As an aside, I'm trying to make this clearer in drafts I'm targeting forwards Media Over QUIC/MOQ, so I actually DO believe you ... Please await results.
A +1 from NAME 's review (URL): Section 6. Evolution of Transport Protocols and Transport Protocol Behaviors I really wonder whether UDP vs. TCP vs. QUIC is actually the relevant distinction. What may actually matter are the transport services provided by the protocol stack (e.g., RFC 8095). I fully agree to a related comment in the INTDIR review.
After looking at this issue. I agree with both NAME and NAME that the emphasis should be on the interaction between (see new section describing this) media transport protocols and the transport services provided by underlying transport protocols. I'm moving the formerly-UDP section below the formerly-TCP section, because it's easier to describe what UDP is NOT doing after describing what TCP IS doing. I'm also moving one paragraph into the formerly-TCP section from Section 6, because it is specific to ABR strategies that assume a reliable transport. More news to come, when I create a PR for this issue ...
We think is now mergrable. Reviewer: Michael Scharf Review result: Ready with Issues This document has been reviewed as part of the transport area review team's ongoing effort to review key IETF documents. These comments were written primarily for the transport area directors, but are copied to the document's authors and WG to allow them to address any issues raised and also to the IETF discussion list for information. When done at the time of IETF Last Call, the authors should consider this review as part of the last-call comments they receive. Please always CC EMAIL if you reply to or forward this review. This informational document surveys network and transport protocol issues that affect quality of experience for streaming applications, in particular video. The overview covers many topics across different technologies. Readers may only be familiar with a subset of these topics and could therefore learn quite a bit from this kind of tutorial. Nonetheless, it remains somewhat unclear what the actual objective of some of the text is. For some topics, quite a bit of technical background is provided, but the discussion is not really comprehensive. In many sections, the document neither derives technical challenges, nor system/protocol requirements, nor practical usage guidance. It is just a kind of tutorial. Quite a bit of text also presents ongoing IETF work, and an RFC with this scope may thus get outdated soon. Section 6.2 deals with topics owned by TCPM. In similar past documents, I have asked for a TCPM presentation prior to an IETF last call in order to ensure that owners of running code are in the loop. I believe this strategy has worked well in the past. Having said this, as TSV-ART reviewer I don't strongly disagree with a publication. All these issues may just be OK for an informational RFC. Some more specific comments: Section 3.2.1. Recognizing Changes from an Expected Baseline This section apparently assumes relatively static path properties, e.g., fixed network connectivity. It would be more useful to analyze the impact of Wifi and 4G/5G networks in one place. Some of this follows later in sections 3.2.1 and Yet, the split between these three sections is unclear. Having a discussion about today's Internet path characteristics and the dynamics at one place could be more useful. The text could also better distinguish between what matters to endpoints and considerations relevant only for middleboxes (such as passive monitors). Section 3.3 Path Requirements The statement "to find the bandwidth requirements for a router on the delivery path" may assume that the bottleneck on the path will be routers. Yet, on a path the bottlenecks could also be a link layer device (e.g., an Ethernet Switch, a Wifi access point, etc.). RFC 6077 (specifically Section 3.1.3) also explains that issue. A better wording may be "to find the bandwidth requirements for a delivery path". Section 3.6. Unpredictable Usage Profiles / Section 3.7. Extremely Unpredictable Usage Profiles I am not fully convinced by the distinction between "unpredictable" and "extremely unpredictable". To me, these two sections could be merged and maybe also be shortened. More specifically, Section 3.7 lists a lot of statistics from the past. What is IMHO a bit missing in Section 3.7 are actual operational considerations. For instance, are there any lessons learnt for the future? Note that Section 3.6 has a reasonable conclusion at the end. Section 4. Latency Considerations and Section 4.1. Ultra Low-Latency I am surprised by the definition "ultra low-latency (less than 1 second)", as well as some of the other numbers. For other real-time communication use cases, "ultra low-latency" would probably imply a latency requirement of the order of one millisecond. For instance, isochronous traffic for motion control in industrial networks may require a latency of 1 ms, and such "ultra-low latency" requirements are discussed elsewhere, e.g., in the DetNet WG. The terminology in this section should be better explained to readers dealing with networked systems with much harder latency requirements. And a reference should be added for these definitions in the context of video streaming. Section 5.5.2 Head-of-Line Blocking If Head-of-Line Blocking is indeed a relevant operational problem, it would be useful to add a corresponding reference (e.g., with measurements). Section 5.5.3. Wide and Rapid Variation in Path Capacity The statement "As many end devices have moved to wireless connectivity for the final hop (Wi-Fi, 5G, or LTE), new problems in bandwidth detction have emerged from radio interference and signal strength effects." could be moved to earlier parts of the document. Quite a bit of new computers apparently only have Wifi connectivity and no Ethernet port, i.e., Wifi may just be their default. Also, wireless may not only be used on the last hop, for instance in meshed setups. This may make the problem even harder. Section 6. Evolution of Transport Protocols and Transport Protocol Behaviors I really wonder whether UDP vs. TCP vs. QUIC is actually the relevant distinction. What may actually matter are the transport services provided by the protocol stack (e.g., RFC 8095). I fully agree to a related comment in the INTDIR review. Section 6.1. UDP and Its Behavior UDP is also used for encrypted tunnels (OpenVPN, Wireguard, etc.). Does encrypted tunneling really have no operational impact on streaming apps other than circuit breakers? (I am thinking about 4K streaming over OpenVPN and the like.) Section 6.2. TCP and Its Behavior This text should IMHO be presented and be discussed in TCPM. Personally, I am not convinced that this document is a good place for discussing the long history of TCP congestion control. If the objective of the document is to provide a tutorial, IMHO it would be more useful to briefly explain the state-of-the-art in year 2022. Some further specific comments on TCP congestion control: It is surprising that RFC 5681 is not referenced.793bis has normative statements regarding congestion control, and most stacks are compatible to what is written in 793bis. Several operating systems use CUBIC by default and support standard Reno. Another general good reference for TCP standards is RFC 7414.Does ECN, DCTCP, etc. really not matter at all, for instance, if a data center hosts video playout servers? Section 6.3. QUIC and Its Behavior As already noted, the discussion about head-of-line-blocking would really benefit from backing by a reference. Section 7. Streaming Encrypted Media As far as I know, streaming users sometimes use encrypted tunnels such as OpenVPN or WireGuard (or IPsec) to access video content. I may miss something, but it is unclear to me how that fits into the categories presented in Section


I actually used the URL in the draft to clone a copy, so I think this is good to go.


Discussion from editor conference call: NAME will update this PR with NAME comments
From NAME (URL) Section 7. Streaming Encrypted Media As far as I know, streaming users sometimes use encrypted tunnels such as OpenVPN or WireGuard (or IPsec) to access video content. I may miss something, but it is unclear to me how that fits into the categories presented in Section (Seems closely related to URL )
The purpose of such tunneling is to circumvent geoblocks and access content from other regions - not to protect the media content. Encrypting the content is done by the content owner for its protection.
So, looking at , I'm seeing this text: I think this means that the effect is that the "transport-level information is opaque to intermediaries" effect described in the document applies, with the additional consideration that intermediaries can't see the IP addresses used inside the tunnel, either. I think the same applies to any IPsec-encrypted tunnel. Do we need to say that? And, if so - can we describe this without using product names and descriptions which may age badly in an RFC?
isn't as easy to read, but I think this is also the rough equivalent of IPsec, from a media operator perspective - all of the transport information and the IP addresses used inside the tunnel are obscured.
Just for the record, definitely worth checking as it is related to this discussion: URL Reviewer: Michael Scharf Review result: Ready with Issues This document has been reviewed as part of the transport area review team's ongoing effort to review key IETF documents. These comments were written primarily for the transport area directors, but are copied to the document's authors and WG to allow them to address any issues raised and also to the IETF discussion list for information. When done at the time of IETF Last Call, the authors should consider this review as part of the last-call comments they receive. Please always CC EMAIL if you reply to or forward this review. This informational document surveys network and transport protocol issues that affect quality of experience for streaming applications, in particular video. The overview covers many topics across different technologies. Readers may only be familiar with a subset of these topics and could therefore learn quite a bit from this kind of tutorial. Nonetheless, it remains somewhat unclear what the actual objective of some of the text is. For some topics, quite a bit of technical background is provided, but the discussion is not really comprehensive. In many sections, the document neither derives technical challenges, nor system/protocol requirements, nor practical usage guidance. It is just a kind of tutorial. Quite a bit of text also presents ongoing IETF work, and an RFC with this scope may thus get outdated soon. Section 6.2 deals with topics owned by TCPM. In similar past documents, I have asked for a TCPM presentation prior to an IETF last call in order to ensure that owners of running code are in the loop. I believe this strategy has worked well in the past. Having said this, as TSV-ART reviewer I don't strongly disagree with a publication. All these issues may just be OK for an informational RFC. Some more specific comments: Section 3.2.1. Recognizing Changes from an Expected Baseline This section apparently assumes relatively static path properties, e.g., fixed network connectivity. It would be more useful to analyze the impact of Wifi and 4G/5G networks in one place. Some of this follows later in sections 3.2.1 and Yet, the split between these three sections is unclear. Having a discussion about today's Internet path characteristics and the dynamics at one place could be more useful. The text could also better distinguish between what matters to endpoints and considerations relevant only for middleboxes (such as passive monitors). Section 3.3 Path Requirements The statement "to find the bandwidth requirements for a router on the delivery path" may assume that the bottleneck on the path will be routers. Yet, on a path the bottlenecks could also be a link layer device (e.g., an Ethernet Switch, a Wifi access point, etc.). RFC 6077 (specifically Section 3.1.3) also explains that issue. A better wording may be "to find the bandwidth requirements for a delivery path". Section 3.6. Unpredictable Usage Profiles / Section 3.7. Extremely Unpredictable Usage Profiles I am not fully convinced by the distinction between "unpredictable" and "extremely unpredictable". To me, these two sections could be merged and maybe also be shortened. More specifically, Section 3.7 lists a lot of statistics from the past. What is IMHO a bit missing in Section 3.7 are actual operational considerations. For instance, are there any lessons learnt for the future? Note that Section 3.6 has a reasonable conclusion at the end. Section 4. Latency Considerations and Section 4.1. Ultra Low-Latency I am surprised by the definition "ultra low-latency (less than 1 second)", as well as some of the other numbers. For other real-time communication use cases, "ultra low-latency" would probably imply a latency requirement of the order of one millisecond. For instance, isochronous traffic for motion control in industrial networks may require a latency of 1 ms, and such "ultra-low latency" requirements are discussed elsewhere, e.g., in the DetNet WG. The terminology in this section should be better explained to readers dealing with networked systems with much harder latency requirements. And a reference should be added for these definitions in the context of video streaming. Section 5.5.2 Head-of-Line Blocking If Head-of-Line Blocking is indeed a relevant operational problem, it would be useful to add a corresponding reference (e.g., with measurements). Section 5.5.3. Wide and Rapid Variation in Path Capacity The statement "As many end devices have moved to wireless connectivity for the final hop (Wi-Fi, 5G, or LTE), new problems in bandwidth detction have emerged from radio interference and signal strength effects." could be moved to earlier parts of the document. Quite a bit of new computers apparently only have Wifi connectivity and no Ethernet port, i.e., Wifi may just be their default. Also, wireless may not only be used on the last hop, for instance in meshed setups. This may make the problem even harder. Section 6. Evolution of Transport Protocols and Transport Protocol Behaviors I really wonder whether UDP vs. TCP vs. QUIC is actually the relevant distinction. What may actually matter are the transport services provided by the protocol stack (e.g., RFC 8095). I fully agree to a related comment in the INTDIR review. Section 6.1. UDP and Its Behavior UDP is also used for encrypted tunnels (OpenVPN, Wireguard, etc.). Does encrypted tunneling really have no operational impact on streaming apps other than circuit breakers? (I am thinking about 4K streaming over OpenVPN and the like.) Section 6.2. TCP and Its Behavior This text should IMHO be presented and be discussed in TCPM. Personally, I am not convinced that this document is a good place for discussing the long history of TCP congestion control. If the objective of the document is to provide a tutorial, IMHO it would be more useful to briefly explain the state-of-the-art in year 2022. Some further specific comments on TCP congestion control: It is surprising that RFC 5681 is not referenced.793bis has normative statements regarding congestion control, and most stacks are compatible to what is written in 793bis. Several operating systems use CUBIC by default and support standard Reno. Another general good reference for TCP standards is RFC 7414.Does ECN, DCTCP, etc. really not matter at all, for instance, if a data center hosts video playout servers? Section 6.3. QUIC and Its Behavior As already noted, the discussion about head-of-line-blocking would really benefit from backing by a reference. Section 7. Streaming Encrypted Media As far as I know, streaming users sometimes use encrypted tunnels such as OpenVPN or WireGuard (or IPsec) to access video content. I may miss something, but it is unclear to me how that fits into the categories presented in Section


looks good.
From NAME
NAME and NAME - I'm not sure who wrote this text, except I don't think it was me. >And as with other parts of the ecosystem, new technology brings new challenges. For example, with the emergence of ultra-low-latency streaming, responses have to start streaming to the end user while still being transmitted to the cache, and while the cache does not yet know the size of the object. Some of the popular caching systems were designed around cache footprint and had deeply ingrained assumptions about knowing the size of objects that are being stored, so the change in design requirements in long-established systems caused some errors in production. Incidents occurred where a transmission error in the connection from the upstream source to the cache could result in the cache holding a truncated segment and transmitting it to the end user's device. In this case, players rendering the stream often had the video freeze until the player was reset. In some cases the truncated object was even cached that way and served later to other players as well, causing continued stalls at the same spot in the video for all players playing the segment delivered from that cache node. NAME was curious if there's a reasonable reference that we could provide for this.
Yeah, that was me. There was not a public reference for this, alas, just a word of mouth anecdote.
NAME - I agree with you about your comment on Section 3.6. It happens that I got comments that Section 3.6 and 3.7 didn't need to be nearly as detailed as they were, so they were heavily rewritten. The text where "throttled" was used is now in Section 3.6.1, and the emphasis is now on the cause of unexpected traffic profile changes, rather than the ISP response to the unexpected traffic profile changes in this specific incident, so the text that included "throttled" has been removed.
NAME - I agree with your comment on Section 4.4, that some description or reference should be provided for the word "manifest". The next occurrence of "manifest" is in Section 5.2, which does provide a description. I don't think that's quite right - I'd say So I'm changing this definition, and moving it to the first occurrence of "manifest" in Section 4.4. Thanks for helping us fix problems you didn't even tell us about. :upsidedownface:
lgtm


Adding Olivier's proposed text about the Fast-Close (cfr., URL) and integrated a shorter version of URL Hello, Sorry for being late with this, but the beginning of the fall semester is always too busy for me. During the summer, there was a discussion on the FASTCLOSE option that was triggered by Francois. The current text of RFC6824bis uses the FASTCLOSE option inside ACK packets, but deployment experience has shown that this results in additional complexity since the ACK must be retransmitted to ensure its reliable delivery. Here is a proposed update for section 3.5 that allows a sender to either send FASTCLOSE inside ACK or RST while a receiver must be ready to process both. Changes are marked with Fast Close Regular TCP has the means of sending a reset (RST) signal to abruptly close a connection. With MPTCP, a reguler RST only has the scope of the subflow and will only close the concerned subflow but not affect the remaining subflows. MPTCP's connection will stay alive at the data level, in order to permit break-before-make handover between subflows. It is therefore necessary to provide an MPTCP-level "reset" to allow the abrupt closure of the whole MPTCP connection, and this is the MPFASTCLOSE option. MPFASTCLOSE is used to indicate to the peer that the connection will be abruptly closed and no data will be accepted anymore. The reasons for triggering an MPFASTCLOSE are implementation specific. Regular TCP does not allow sending a RST while the connection is in a synchronized state [RFC0793]. Nevertheless, implementations allow the sending of a RST in this state, if, for example, the operating system is running out of resources. In these cases, MPTCP should send the MPFASTCLOSE. This option is illustrated in Figure 14. 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +---------------+---------------+-------+-----------------------+ +---------------+---------------+-------+-----------------------+ +---------------------------------------------------------------+ Figure 14: Fast Close (MPFASTCLOSE) Option If Host A wants to force the closure of an MPTCP connection, it has two different options : Option A (ACK) : Host A sends an ACK containing the MPFASTCLOSE option on one subflow, containing the key of Host B as declared in the initial connection handshake. On all the other subflows, Host A sends a regular TCP RST to close these subflows, and tears them down. Host A now enters FASTCLOSEWAIT state. Option R (RST) : Host A sends a RST containing the MPFASTCLOSE option on all subflows, containing the key of Host B as declared in the initial connection handshake. Host A can tear the subflows and the connection down immediately. If a host receives a packet with a valid MPFASTCLOSE option, it shall process it as follows : o Upon receipt of an ACK with MPFASTCLOSE, containing the valid key, Host B answers on the same subflow with a TCP RST and tears down all subflows. Host B can now close the whole MPTCP connection (it transitions directly to CLOSED state). o As soon as Host A has received the TCP RST on the remaining subflow, it can close this subflow and tear down the whole connection (transition from FASTCLOSEWAIT to CLOSED states). If Host A receives an MPFASTCLOSE instead of a TCP RST, both hosts attempted fast closure simultaneously. Host A should reply with a TCP RST and tear down the connection. o If Host A does not receive a TCP RST in reply to its MPFASTCLOSE after one retransmission timeout (RTO) (the RTO of the subflow where the MPTCPRST has been sent), it SHOULD retransmit the MPFASTCLOSE. The number of retransmissions SHOULD be limited to avoid this connection from being retained for a long time, but this limit is implementation specific. A RECOMMENDED number is 3. If no TCP RST is received in response, Host A SHOULD send a TCP RST with the MPFASTCLOSE option itself when it releases state in order to clear any remaining state at middleboxes. o Upon receipt of a RST with MPFASTCLOSE, containing the valid key, Host B tears down all subflows. Host B can now close the whole MPTCP connection (it transitions directly to CLOSED state). The burden of the MP_FASTCLOSE is on the host that decides to close the connection (typically the server, but not necessarily). The text above allows it to opt for either ACK or RST without adding complexity to the other side. Another point that might be discussed in RFC6824bis is how a host should react when there are all the subflows associates to a Multipath TCP connection are in the CLOSED state, but the connection is still alive. At this point, it is impossible to send or receive data over the connection. It should attempt to reestablish one subflow to keep the connection alive. Olivier