Note that I do need you both to confirm that you have fulfilled IPR obligations.
Thank you!


Connection Migration .... " it sends or it can simply send an IMMEDIATE_ACK frame" What is the intended difference between the words "it sends" or "it can simply send"?


Provides more details on why excluding maxackdelay when Ignore Order is used may cause premature PTOs.
I'm merging this -- I think this is good, but if we need to add more cautionary text, we can still do that.
When the number of in-flight ack-eliciting packets is larger than the ACK-Eliciting Threshold, an endpoint can expect that the peer will not need to wait for its maxackdelay period before sending an acknowledgement. In such cases, the endpoint MAY therefore exclude the peer's 'maxackdelay' from its PTO calculation. Note that this optimization requires some care in implementation, since it can cause premature PTOs under packet loss when ignoreorder is enabled. Isn't this oversimplifying things so that the aspect of time until an ACK is expected to be sent is removed? To me it is not that maxackdelay is removed completely from URL PTO = smoothedrtt + max(4*rttvar, kGranularity) + maxackdelay Instead it is replaced with expected time until the sender have sent ACK-Eliciting Threshold number of packets when that is less than maxackdelay that is configured. Secondly, are you expecting the sender to make an estimate of how many packets more are received before an ACK is sent, i.e. care about what phase it is currently when calculating the PTO in regards to which packet is expected to trigger the next ACK? However, that is clearly prone to be wrong if there is a loss occurring currently who's loss indicating ack has not yet been received by the sender. So it appears to me that this allowance is by default creating a situation which will cause premature PTOs, and it would be better to put some restrictions on it.
On the second point, the sender always knows how many packets are outstanding. So, it always knows whether the PTO computation -- which is always on the latest sent packet -- should include maxackdelay or not. That said, you are right the ignoreorder causes trouble here, so how about "MAY therefore exclude the peer's 'maxackdelay' from its PTO calculation unless ignoreorder is enabled. When ignore_order is enabled, premature PTOs are significantly more likely under packet loss."


I tried to work in what I thought were the important parts of Gorry's suggested text, PTAL if I captured everything.
It's also true that for many RF media, the performance of the forward path can be impacted by the rate of acknowledgment packets when the link bandwidth is shared.
“Severely asymmetric link technologies, such as DOCSIS, LTE, and satellite links, connection throughput in the data direction becomes constrained when the reverse bandwidth is filled by acknowledgment packets. When traversing such links, reducing the number of acknowledgments allows connection throughput to scale much further.” I’m going to push back again on this as largely wrong. Links such as DOCSIS, LTE, satellite and WiFi are asymmetric. But it’s not just that the design decisions made for these types of link resulted in asymmetric capacity (throughput limit), the designs also share the capacity, and it is important to mitigate the impact on that sharing of capacity, than purely thinking of it items of bytes/sec of capacity used. A lower feedback rate can help avoid inducing flow starvation to other flows that share resources along the path they use.On mobile devices sending data consumes RF and battery power and is more costly for a host of reasons not just asymmetry when present.
NAME -- I don't follow why you say that the statement is wrong. It seems to me that you think that the statement is inadequate. Would you mind responding here with some text you would prefer instead, and I'm happy to wordsmith and turn it into a PR?
I suggest some text something like: For some link technologies (e.g., DOCSIS, LTE, WiFi, and satellite links) transmission of acknowledgment packets contribute to link utilization and can incur significant forwarding costs. In some cases, the connection throughput in the data direction can become constrained when the reverse bandwidth is filled by acknowledgment packets. In other cases, the rate of acknowledgment packets can impact link efficiency (e.g. transmission opportunities or battery life). A lower ACK frequency can be beneficial when using paths that include such links.
NAME -- Is the text proposal in the issue enough for you to make a PR?
This is much better.


Sec #ack-frequency-frame says: and Sec #out-of-order says:
It seems optimal to default it to 3, because that's the default packet threshold, but the principle of least surprise argues for 1 (ie: no behavior change until an ACK_FREQUENCY frame is received), so I lean toward 1.
This PR adds a SHOULD NOT, but it seems sensible.


We have inconsistent definitions for Ack-Eliciting Threshold, which probably has people implementing this differently. In Section 4: In Section 7: The intended definition is the one in Section 4, so fixes and makes this consistent.
The document uses MUST in multiple places, however, as discussed in the new proposed text for the security considerations (see PR ) the requester cannot enforce any ACK strategy and there might be reason like practical limitation or an overload situation where the ACK sender simply decides to send less ACKs. So the question is really can to requirements be MUSTs or is SHOULD more realistic and correct?


This PR changes a few things: Introduces a TP for negotiating extension use Changes to Makes the fields required Specifies invalid values
I'm merging this, let's address issues in later PRs.
I think it'll be easier to make both parameters required. Not doing so saves 1 to 2 bytes, which is really not much in a frame I expect will be sent a few times a connection. Also, I think 0 should be an invalid value for both params, and 1 should be invalid for the number of packets, though I can see someone arguing that point.
addresses these issues, with the exception that 1 is allowed for packet tolerance. I think that's actually useful for some startup schemes, so we shouldn't disallow it.


This uses two JWS signatures to achieve the two-signature property during rollover, instead of using extra layers of encoding and embedding. It also simplifies the logic and splits it out into its own resource.


what's the difference between "deactivated" and "revoked"?
Deactivated is done by the user; revoked is done by an administrator.
OK, so we need to either say that or use clearer terms :) I talked with NAME offline, and I think he's going to propose something.


Based on the acme mailing list discussion[0] on "proactive" vs "on-finalization" issuance it seems there is more support for the proactive approach. Speaking on behalf of Let's Encrypt and the Boulder developers we're willing to compromise and support proactive issuance as the sole issuance method in order to simplify the protocol & implementations. This PR changes the language around certificate issuance to reflect that the server MUST issue the certificate proactively once all the required challenges for an application have been fulfilled. The open issue corresponding to making this decision is removed now that there is one supported option. In an effort to close out open questions around applications this commit also removes the open issue around multiple certificates per application. If there is a demand for automating repeat issuance there should be a new thread started with a concrete proposal. 0: URL
NAME apologies! I missed your comments until today. Both are resolved in URL Thank you for the feedback.
Oh! They were fresh and I misread the dates. Not such a bad turn around after all :-)
NAME Fantastic, thanks. Merging. Hi folks, I wanted to revisit a discussion on the server proactively issuing certificates. Section 4, "Protocol Overview"[0] currently has a paragraph that reads: the the application be to the There is also an "Open Issue" remark related to proactive issuance in Section [1]. I think the specification should decide whether proactive issuance or on-finalization issuance should be used instead of allowing both with no indication to the client which will happen. It's the preference of Let's Encrypt that the specification only support on-finalization issuance. This is cheaper to implement and seems to be the path of least surprise. It also seems to better support large volume issuance when the client may want explicit control over when the certificates for multiple applications are issued. In the earlier "Re: [Acme] Preconditions" thread[2] Andrew Ayer seems to agree that there should be only one issuance method to simplify client behaviours, though he favours proactive issuance instead of on-finalization issuance. While it saves a round-trip message I'm not sure that this alone is convincing enough to choose proactive issuance as the one issuance method. What are the thoughts of other list members? Daniel/cpu [0] URL [1] URL [2] URL


In the term "registration" was changed to "account". In and in some of the remaining references to "registration" were removed, but some 'new-reg' and references to "registration" remained. This PR removes, hopefully, the last references to "registration".


This PR is a first go at addressing the following points I raised on the mailing list.
This allows us to address the account URL recovery use case in without having special handling for empty payloads.


This change replaces "shortly" in the normative language of Section 7.4, Applying for Certificate Issuance, with the more precise expectation that the server begin the issuance process, as discussed on the mailing list. URL We could say "MUST begin the issuance process" The main things on my mind that could delay issuance slightly: Submitting to CT Checking CAA Internal queuing for available capacity Manual vetting I think "MUST begin" covers for all of those, while allowing some vagueness as to how long they will take. On 03/22/2017 09:39 AM, Daniel McCarney wrote:


Is 200 really correct? This seems like a 404 maybe? Also we should specify the body (empty).
Would a problem document be appropriate (with either a pre-existing ACME error type or a new one like "accountDoesNotExist")?
404 talks about the effective request URI, so that's not going to work. I think that a 400 would be better on the basis that the request checked, but failed (NAME suggestion is a fine enhancement). The reason you make this request is to check; and having a 4xx series code is easier to test than the absence of a header field.
Thanks NAME and NAME implemented this in the latest commit.
As discussed; this is fine.


It makes sense to cancel any pending cert orders when an account is deactivated, where the cert has not been issued yet.


Previously there was some notion of being able to hint to the server whether you wanted to reuse authorizations. This was removed from the spec but the pre-authorization flow still made a reference to it. This commit removes that obsolete reference. Thanks to NAME for pointing out the error. Resolves URL
Hi in the section on pre-authz there is this sentence: To request authorization for an identifier, the client sends a POST request to the new-authorization resource specifying the identifier for which authorization is being requested and how the server should behave with respect to existing authorizations for this identifier. the part "and how the server should behave with respect to existing authorizations for this identifier" probably is obsoleted
Hi NAME I submitted URL to fix this. Thanks for spotting & reporting :-)


I think without OOB all challenges use key-authorization "JWA" is not defined in this document. I think it can just be omitted
NAME Thank you for the change! :smile:
Thanks again NAME I really appreciate your editing & the accompanying PRs! :-)


Prior to this commit the literal string "." was being rendered as the start of italics in the generated HTML doc. This commit updates the markup to escape the "*."` in the generated document. Originally reported by Joel Sing - thanks!


In Section 7.4.1 "Pre-Authorization" the spec says: If a CA wishes to allow pre-authorization within ACME, it can offer a "new authorization" resource in its directory by adding the field "newAuthz" with a URL for the new authorization resource. That text indicates that the CA may wish to not support pre-authorization in which case the "newAuthz" resource will not be present in the directory. One example of an ACME CA choosing to not support pre-authorization is Let's Encrypt. This commit clarifies the "newAuthz" field of the directory is optional in Section 7.1.1 "Directory" where a developer is likely to look to understand which resources must be supported/present in the directory and which are optional ("meta", "newAuthz"). If the server doesn't support pre-authorization it MUST omit the "newAuthz" field.


In addition to adding the error, as proposed in , this PR fixes some issues discussed in the F2F meeting, namely moving ACME URNs under the IETF parameters space, and having errors be a sub-namespace of an overall ACME namespace.


URL
Thanks NAME On Sat, Sep 15, 2018 at 08:51:55PM -0500, Benjamin Kaduk wrote: My apologies if I missed it when it went by, but did we ever hear more about this requirement from the proponents of externalAccountBinding? Thanks, Benjamin


Some grammar fixes POST-as-GETs should be indicated as having a signature in flow Specify that the JWS profile we define in Section 6.2 is for ACME request bodies, to distinuish it from the profile used for External Account Binding. They have incompatible requirements: MUST NOT use MAC vs MUST use MAC. Clarify that Replay-Nonce is an HTTP header field since it's defined near JWS header parameters. Don't include "wildcard": false in examples. Section 7.1.4 requires that it MUST be absent when false. There was lingering language in the Section 7.5.1 about "updating" an authorization body in response to fields in a challenge POST. Since the challenge POST is now described explicitly as "an empty JSON body", this language didn't make sense. Fixed.
Per offline review, I added back in a sentence that contained a MUST:


On the mailing list, NAME and NAME pointed out some risks arising from the default virtual host behavior of some web servers. This PR integrates the changes they suggest, namely (1) removing the "tls" option from the challenge, and (2) adding iterations to the challenge. This PR is based on top of and .
Sorry, feel free to tell me to go away, but reading the backing decision to drop HTTPS for SimpleHTTP doesn't make sense to me. If I understand everything correctly, you control the originator of the HTTPS requests and the requests are always sent to domains. So why can't you just enforce to always send the Host header and this wouldn't be an issue?
From : behaviour that if a default TLS/HTTPS virtual host is not configured explicitly, one will be selected based on the ordering of vhosts in the webserver configuration (in practice, this often amounts to alphabetical ordering of files in a configuration directory). As far as I know Apache does the exact same thing for nonencrypted HTTP hosts (this is also acknowledged by a later mail in the thread), so the reasoning is a little weird. I suppose the idea is that hosters are more likely to have HTTP configured correctly rather than HTTPS?
The section on the "" registry says that "Fields marked as "client configurable" may be included in a new-account request". This seems like a copy-paste-o, and perhaps should say that such fields may be included in a new-order request.
Good catch. I opened URL to fix. Thanks! Apache (and I gather nginx too) have the subtle and non-intuitive behaviour that if a default TLS/HTTPS virtual host is not configured explicitly, one will be selected based on the ordering of vhosts in the webserver configuration (in practice, this often amounts to alphabetical ordering of files in a configuration directory). URL URL This creates a vulnerability for SimpleHTTP and DVSNI in any multiple-tenant virtual hosting environment that failed to explicitly select a default vhost [1]. The vulnerability allows one tenant (typically one with the alphabetically lowest domain name -- a situation that may be easy for an attacker to arrange) to obtain certificates for other tenants. The exact conditions for the vulerability vary: SimpleHTTP is vulnerable if no default vhost is set explicitly, the server accepts the "tls": true parameter as currently specified, and the domains served over TLS/HTTPS are a strict subset of those served over HTTP. DVSNI is vulnerable if no default vhost is set explicitly, and the hosting environment provides a way for tenants to upload arbitrary certificates to be served on their vhost. James Kasten and Alex Halderman have done great work proposing a fix for DVSNI: URL That fix also has some significant performance benefits on systems with many virtual hosts, and the resulting challenge type should perhaps be called DVSNI2. The simplest fix for SimpleHTTP is to remove the "tls" option from the client's SimpleHTTP response, and require that the challenge be completed on port 80. URL Alternatively, the protocol could allow the server to specify which ports it will perform SimpleHTTP and SimpleHTTPS requests on; HTTPS on port 443 SHOULD be avoided unless the CA can confirm that the server has an explicitly configured default vhost. The above diffs are currently against Let's Encrypt's version of the ACME spec, we'll be happy to rebase them against the IETF version upon request. [1] though I haven't confirmed it, it's possible that sites that attempted to set a default vhost are vulernable if they followed advice like this when doing so: URL ; an attacker need only sign up with a lexicographically lower name on that hosting platform to obtain control of the default. Peter Eckersley EMAIL Chief Computer Scientist Tel +1 415 436 9333 x131 Electronic Frontier Foundation Fax +1 415 436 9993


Included in the text. Maybe will include further text for
In 5.5.1: This kind of suggests that an Agent may also not know the serial number. But in creating the agent-signed-data, this number is required later on. Could we clarify perhaps the Agent MUST know the serial-number; e.g. in one of these ways: configured (as in quoted text above) discovered (in "standard" way using mDNS / DNS-SD first part of service instance name) discovered (in vendor-specific way e.g. RF beacons) other (in vendor-specific way, e.g. scan QR code of packaging) - maybe that's the same as category 1 from the Agent point of view.
Additional Note: figure 4 doesn't show the part where the Agent discovers the serial number of the Pledge. This was adding to the confusion. Maybe figure 4 can stay as is; this is just to clarify that at the start of Fig 4 the Agent MUST know the serial-number. So some small link is missing to get the complete picture.
I reopened as the initial point if the registrar-agent MUST know the serial number is not solved by the QR code reference. As the registrar-agent needs to know, which pledges he is bootstrapping, he needs the list of serial numbers in advance to be able to construct the agent signed data. Text proposed will be included into the pre-conditions described in section 5.5.1, In addition, the registrar-agent MUST know the product-serial-number(s) of the pledge(s) to be bootstrapped. The registrar-agent MAY be provided with the product-serial-number in different ways: configured, e.g., as a list of pledges to be bootstrapped via QR code scanning discovered by using standard approaches like mDNS as described in 5.4.2. discovered by using a vendor specific approach, e.g., RF beacons
Ok


It seems blindingly obvious to me that an implementation would need to take care of this... and this note of caution seems quite appropriate (if unnecessary... but surely it doesn't hurt).
Dynamic framer add and passthrough (last two paragraphs of section 6.1) assume synchronous operation; what happens to bytes in the send/receive buffers when the framer stack changes? There either needs to be an explicit mutex here (allowing the framer/implementation to lock the buffers at the exact point the framer change happens, not allowing any bytes to be processed while the dynamic framer decision is made), or some way to annotate in a buffer the replay point after which a dynamic reframing is necessary. Are we going to leave this up to implementors to figure out, or should we point out that here be dragons?
We should clarify that these operations within the data path of the stack should be synchronized.


SORRY I made a PR on your PR, when I meant to leave a review comment, see 1024. .... Silly to make a PR on a PR - I added a review comment instead...
In section 4.1.4 in the impl draft, we have: "The order of operations for branching, where lower numbers are acted upon first, should be: Alternate Paths Protocol Options Derived Endpoints" However, I would say there is not really a strict ordering required. The text actually indicates that selection of the protocol is rather independent while there is usually a dependency between the interface and derived endpoint. I think this is related to my other issue about the use of the term path, because a path consists of both the local and remote endpoint and therefore there is of course a dependency. So in summary I'm not sure how useful I find this whole section as currently written...
We should conclude if anything more is needed here besides cleaning up the terminology, where Alternate Paths is not a good term - see .
Any implementation should have a consistent order of operations. This is a recommended example. Reference the arch text that says it needs to work the same way each time.
Reference ICE/HE


NAME wrote: "Why is the text talking about "many properties" that can no longer be changed, instead of relying on the earlier, nice and clear distinction between Selection and Connection Properties?". => In line with the suggestion from NAME this is now changed, it says Selection Properties. NAME wrote: "Where earlier, Selection Properties were said to remain readable, now they are only queryable?". => In line with NAME answer, this PR replaces "query" with "read" in this text. NAME wrote: "Are "applied" Selection Properties still Selection Properties? Shouldn't they better be referred to using a different term?" and here, NAME suggested: "In the , we define Protocol Stack as a Transport Services implementation concept, and we talk about a certain protocol stacks providing certain features (see . Perhaps we could rephrase this part of the API draft to use that terminology, e.g., the application can query the actually used protocol stack and then know which Transport Features it supports." => I don't know how to apply this change, and it generally sounds a bit complicated to me. I think that Selection Properties should keep their name. Regarding the confusing text with the example, I removed this and I think it makes things clearer. Regarding the return type, I think I now finally understood the bigger problem. Returning a Boolean instead of a Preference should make sense and should be easy to implement (as also described in this PR). The problem is that not all Selection Properties are of type Preference. E.g., is an enumeration, representing , or . What sense does it make if this becomes only or ? Similarly, and are of type "Collection of (Preference, Enumeration)", and indeed it would make sense to expect a Collection of booleans, not a singular boolean. This PR now explicitly says that, upon reading, the Preference type of a Selection Property (not the whole Selection Property!) changes into Boolean.
MANY thanks for the proposed fix (which I committed)! I hadn't really understood the problem just from reading the issue discussion.
As , here follows a bit of feedback on the different Property types in the Interface draft, from the perspective of an academic developer who is currently attempting to follow the draft spec as closely as possible while trying to implement a path-aware transport system in a In , it says: At this point, the reader of the document could be expected to conclude the following: Selection Properties are a set of preferences for protocol/path selection that (obviously) can no longer be meaningfully changed once a connection has been established. Connection Properties are a set of "run-time preferences" for connections which could still be adjusted when a connection is already established, but should ideally already be known during protocol/path selection The text goes on to state: With this wording, the reader can now reasonably conclude that the original Selection Properties that have resulted in a given Connection should remain available during the lifetime of that Connection, perhaps in form of a copy of the original data structure. So far so straight forward. In however, this conclusion is called into question. After the text has gone on to introduce the 5 different Preference Levels, it states At this point, the reader becomes confused. Why is the text talking about "many properties" that can no longer be changed, instead of relying on the earlier, nice and clear distinction between Selection and Connection Properties? Where earlier, Selection Properties were said to remain readable, now they are only queryable? The type of a Selection Property that has been "applied" mutates from a Preference Level to become a Boolean? Are "applied" Selection Properties still Selection Properties? Shouldn't they better be referred to using a different term? So the original Selection Preferences that resulted in the current Connection are not available after all? Only the (Boolean) Properties of the chosen Protocol can be "queried"? But wait, the text goes on: This is a complicated way of saying that the values only reflect the actual Properties of the chosen Connection and (in direct contradiction to the previous paragraph) do not actually indicate whether a Selection Property has also been "applied" or not. (A Boolean only carries exactly one bit of information after all.) In other words, the Protocol Properties can be queried. The usefulness of this is probably limited however. If we already know that our Connection is using, e.g., TCP, we don't have to issue any "queries" to know that the Connection offers reliable transport, is subject to congestion control and preserves packet ordering. (This is indeed how I have chosen to implement this. The chosen Protocol can be queried for the Properties it inherently has. In the Preconnection phase, the Selection Preferences are matched against these and any / conflict results in that Protocol being dropped from consideration.) Hoping not to be unreasonable (or somehow block-headed), I have the following suggestions Either introduce a new term for Properties that have been "applied", or Use the term "Preferences" (e.g., "Selection Preferences", "Connection Preferences") to refer to a collection of adjustable "Settings" and the term "Properties" to refer to measured or queried values that are actual Properties of underlying Connections and Protocols Go into a bit more detail about what the draft really means when talking about "querying" Properties (versus setting them), maybe even in terms of exemplary interactions with Property data structures. (After all, the draft elsewhere goes into considerable detail with it's exemplary Event-based system) Think about whether it in general would clear things up when the explicit notion of Protocol Properties as a (static) counterpoint to (adjustable) Selection Properties is introduced
(Replying as an individual and co-author) Thanks for raising this issue! Yes, this matches my understanding is well. If the later text confuses this issue for the reader, then perhaps we'll need to tweak it. I'd say that Selection Properties can no longer be changed, Connection Properties still can be changed (though maybe changing a Connection Property later would not have any effect, e.g., if Capacity Profile changes but was only taken into account when selecting a path). Perhaps the text should reflect that. For me, these are synonyms, whereby a "query" is what the application has to do in order to read the properties. Perhaps the "queried" should become "read" to avoid confusion. That is indeed how I think of it on a conceptual level, but I understand how this might confuse implementers, especially if they can't just change a data type of a property. In the , we define Protocol Stack as a Transport Services implementation concept, and we talk about a certain protocol stacks providing certain features (see . Perhaps we could rephrase this part of the API draft to use that terminology, e.g., the application can query the actually used protocol stack and then know which Transport Features it supports. I would strongly prefer using our existing terminology to introducing yet another term. Good question. I vaguely remember discussing this, and I think folks felt like querying the Selection Properties that the application had set was not very useful at that stage. The other text you quoted indeed sounds a bit complicated and maybe it can be reworded to be clearer. Curious to hear what my co-authors and/or other TAPS participants think!
I completely second what NAME wrote. With regards to querying the Transport Features selected vs. getting the preferences provided earlier, I remember we had lengthy discussions and somewhat converged to the current state. Primary arguments ware Multiple read/query variants are confusing Original Preferences are not really useful - if you really want to read them later, keep the Preferences Object Constrained implementations don't want to be forced to store them. NAME based on these arguments, what option would you prefer a) different get/query operations of selection properties after the connection has been set up b) the current way of "mutating" preferences with clearer description in the text
Just to add: having finally gotten around to carefully reading this, I also completely agree with everything NAME wrote. This is exactly how I would have answered, and her suggestions are exactly what I think should be done to address these comments. I also agree with NAME that these were the arguments regarding querying the querying issue, and I agree with the arguments as well. The quoted text containing the example is a great catch! Such a terribly convoluted paragraph. I have a nagging feeling that this was my fault! :-(
PR tries to solve this; the PR itself contains a full explanation of my rationale for the changes. Please take a look!
Great, this looks good to me now. I think now it is simple enough that we don't need an example anymore.Looks good, one clarification suggestedWorks for me (with NAME changes). I would propose a slight rewording -> and add a hint that this exposing can also be implemented differently.


Appendix B.1 lists two properties as "under discussion": Bounds on Send or Receive Rate: (PR 1036 fixes the name): this is no longer under discussion, these properties exist. Cost Preferences: this is no longer under discussion, I suppose?? We don't have this in the interface draft. So I have the impression that this should no longer be an appendix, the text on Cost Preferences should be removed, and the text on Bounds on Send and Receive Rate should be moved elsewhere, and not be mentioned as "under discussion".
LGMT


Capitalization seems to be a problem for us. We need to make things uniform. At first, I reacted to seeing "connection" as parameters in this PR, because we use the term "Connection" to refer to a TAPS-provided connection (as opposed to a connection of the underlying system), and so, de-capitalizing the term here may look better, but it is also somewhat imprecise. This said, we do the same with other capitalized terms too, here and in the API draft - e.g., "messageContext". Should we generally agree to begin terms with a lowercase letter when they are used as parameters in code examples? I think we have this in most places, though at least the API draft has several counterexamples... e.g., the server example code: Preconnection := NewPreconnection(LocalSpecifier, TransportProperties, SecurityParameters) (..) Listener -> ConnectionReceived
From discussion on
This was a mistake. As I wanted to do this now, I noticed that all occurrences of capitalized terms as parameters are objects that are defined and used elsewhere in the code examples. For example: Preconnection := NewPreconnection(RemoteSpecifier, TransportProperties, SecurityParameters) refers to RemoteSpecifier etc., being objects that are initialized some lines above. Making them appear in lower case everywhere would be awkward, this is then a much more drastic change to our pseudocode. As a result, I think it's fine to leave things as they are, and I suggest to close this issue with no action.
Framers: What is the difference between the Final message property and the separate IsEndOfMessage event argument?
Also there, the term MessageContext could use a definition
and searching "isEndOfMessage" tells me that there is no text describing this parameter. Regarding the question, the difference should be clear: seems self-explanatory, whereas indicates that the Message carrying it is the last one of the Connection. So, this is End of the Message vs. End of the Connection.
Sounds like this needs a reference to the API doc, and change IsEndOfMessage to endOfMessage to match the API.
Probably fix with
I think this looks ok (and thanks for doing this!), but see also my comment about making things uniform wrt. capitalization: if we merge this PR, we should also do one that does the same with the API draft.Thanks LGTM! Although I noticed another nit that can perhaps be fixed in the same PR even if not directly related to the parameter names. Regarding the capitalization I think we can merge this PR and then have a separate issue for the capitalization as we will need to fix this either way.


Takes care of item 5 and .
See discussion on
Provide a complete template that covers all functions/properties from the API. Indicate how the template should be used (e.g. leave out the parts that are not relevant for a particular protocol or include all fields and mark fields that are not relevant as such.
I'm wondering if this is really something that's in scope that we should cover. We do have the message properties section that explains how to implement various properties, but I think adding each property to every protocol mapping will be a huge amount of text that will be difficult to get exactly right.
I agree, we have added additional properties since we opened this issue to check the mapping and make sure we have a complete coverage. We should still be consistent in how we treat things though, and perhaps add a sentence that we do not describe all the properties for instance. I have made a pass through to check the mapping from the API to impl. draft for all the protocol mappings as a basis for making things consistent. Here is the summary. We cover the same actions and events for all the protocols with the exceptions that UDP Multicast does not cover the ConnectionError event and SCTP covers the Priority connection property. And MPTCP and UDP-lite are special as they refer back to the base protocols. There is one terminology inconsistency where the impl. draft refers to InitiateError and this has been generalized to EstablishmentError in the API Unless I missed something, there are only two actions that can be made on a connection that are not covered, addRemote and removeRemote. This is probably fine as they do not mean much for most of the protocols (or we could add them to be complete). We should perhaps mention how they relate to MPTCP though if we leave them out. We also do not cover the Preconnection.Rendezvous action. Again this may be fine as it is a bit special (or we can mention it for consistency.) For the events, we cover a subset with main focus on the connection related events. The Closed Event and some events related to connection setup are missing. As we cover ConnectionError we should perhaps cover the Closed event as well? We do not cover any of the Send or Receive events which seems fine. In relation to Connection and Message properties, we do not cover any of them except for the Connection priority covered for SCTP and we talk about some properties in connection to UDP-lite and MPTCP. SCTP also has support for partial reliability and unordered messages so not sure why the priority part was singled out and added. NAME any particular reasoning behind this part? Possibly we should remove the priority part for SCTP or discuss it in running text in a similar fashion as for UDP-lite and MPTCP We may want to update the UDP-lite description to also say something about what properties it supports over UDP. The current description sounds like it is a subset. Once we agree on how to deal with the bullets above, only smaller modifications should be needed to the text.
ConnectionError should be added; SCTP priority should be removed Fix the inconsistency Add addRemote and removeRemote text related to MPTCP No action Mention Closed Event See item 1, no further action Update UDP-Lite description


We don't have "Protocol Properties" - two occurrences in -interface should really refer to "protocol-specific Properties", and two occurrences in -impl should really refer to properties.
URL Are Protocol Specific Properties and Protocol Properties same or different? Also note that we are first time defining the Protocol Properties in the interface document ( described along with Connection Properties) and not introduced in the architecture document. I think we should be clear about what we are defining where and describing where. I would suggest use of cross reference rather describing everything again and again, which might create confusion and inconsistency.
Oh, this seems like a mistake (great catch, thanks!) - "Protocol Properties" don't exist, I think we only have "protocol-specific Properties".
I caught this when reading the implementation doc. So it should be fixed there as well.


URL Assuming there would be more protocols in the future and the minset functionalities will expand via extension or amendments. It would important to indicate if this architecture should is also extensible or it only supports on the minimum set referred. Definitely we haven't seen the future yet, but it would be important to express if we have considered such expansion of minimum functionalities.
To me, the current spec is based on minset that describes the common functions. So, the addition I would expect here is to say that there can be extensions and protocol-specific features.
I expect that if a common feature is added to the minset, then the architecture and API surface would also be extended to match.
LGTM modulo a nit


URL Unless there is a description the figure, with directed and non-directed lines it is very hard to interpret the figure. I would recommend to at least add some reading guidance for the lines.
LGTM, thanks!Looks fine.


(sorry that it took me so long) This uses some text that is also in the interface draft.
This I would like us to discuss. Well I know we have discussed this before. As I am reading the Interface document after a long while and having read it along with the architecture document, I kind of strongly feel that the architecture document lack of description on framer and the states. These are mentioned and pictured in the architecture document but haven't been clear there. Hence, I kind of feel that we need to move this two sections to architecture document. Please discuss so that I can understand the reasoning of not doing so.
Regarding section 9.1.2, the framer description: I think it's good to be clear and complete in the API document, but I agree that framers should be appropriately described in the architecture document. I have the impression that the text in the -arch document on this matter may indeed be too short, while the text in the -interface document may be too long (although a little bit of repetition surely won't hurt here, it's good for this document to be understandable and clear on its own). Maybe some of it could be moved to -arch, or repeated there? Regarding section 11, Connection states: to me, this is fine as it is. Section 4.1 / figure 4 in the architecture document make state transitions pretty clear - and section 11 / figure 2 are just a more fine-grain look at these states as a way to explain exactly which Events will occur when. This seems needed in the API document, where all other Events and Actions are defined, and I would think that this is at a too detailed / fine-grain level for the architecture document.
NAME after giving it some more thoughts, I think your suggested approach should be fine , meaning we put more description of framer in the architecture document and leave the connection states to the Interface document.
NAME did you want to take a stab at this?
Looks great!


Feel free to approve or propose changes to improve this sentence in Arch.
BT: "I’m not convinced of the value of the glossary of terms, as it forwardrefs basically the entire document, and some of the definitions are so generic as to be potentially confusing (e.g. Event), so I think putting these up front reduces the document’s approachability and readability. I understand that some people like these (and other standards orgs make them mandatory, hi ETSI), so I won’t fight it." "Adding a sentence making it clear it’s safe to skip the glossary if you’re going to read the rest of the document would address my approachability concern completely. "


This closed As we have no methods on a connection object that query properties, cansend/canreceive also became connection properties. As with all the other connection properties, these are indeed a little underspecified. I want to keep it that way till we fix it for all connection properties with and
Aehm… that comes surprising – NAME why didn't you bring this up in ? Do you think this pull-request is the right place to discuss this issue? I would prefer we merge it now and make a new issue for the question whether the uni-directional connection should be more reflected in the connection. I am not particularly found on keeping it a selection property but would like to have the concept of unidirectional connections rather sooner than later. Our consensus in was that unidirectional connection should use the same abstraction for QUIC unidirectional streams, multicast sources/sinks, and half-closed connections – despite they are constructed in completely different ways. The Selection property is just for the QUIC style ones…

Aehm… that comes surprising – NAME why didn't you bring this up in ? Do you think this pull-request is the right place to discuss this issue? Happy to move the discussion back over to the issue if you like. Will continue the thread here for now though since you've got points to address below. fine, but keep open then, because adding the selection property is possibly-necessary but definitely-not-sufficient to fully address unidirectional connection semantics. good, I agree with this. then I'm confused why this PR purports to since it only addresses part of the issue...?
NAME agreed. The question (and discussion) was about support for including some support for unidirectional connections (with focus on QUIC and half-closed), but omitted details on multicast, as this has not laned yet. So indeed, this pull-requests does everything requested by , but does not fully capture the topic. I will move the open issues (Multicast support, is the Selection Property enough, find some place for more explanation of Uni/Bidi Streams) to once this PR is merged.
In Section 9 – "Setting and Querying of Connection Properties" – the properties of an established connection and path are heavily underspecified. Should be fixed along with .
I'm working on this issue within our pull request for , using the terminology as simplified there. For Section 9, I'm proposing the following: An application can query Connection Properties on a connection. Connection Properties include the status of the connection and whether it is possible to send and to receive data on it. (One of them might not be possible, e.g., in a unidirectional connection.) Additionally, Connection Properties include the Connection's Transport Properties. Transport Properties include Selection Properties, so the application can know which Protocol Selection Properties had been specified, and which ones are actually supported by the protocol that has been chosen. For Establishing connections (i.e., we are not done racing yet, the final protocols have not been selected yet), the application gets the Selection Properties that had been specified on the Preconnection earlier. For Established, Closing, or Closed connections (i.e., the protocols have been selected), the application learns whether the Selection Properties are actually provided by the selected protocols. For example, if the application specified earlier, "Control Checksum Coverage: Prefer", now it will get "Control checksum coverage: True", indicating that on this connection it can indeed control the checksum coverage. Transport Properties also include Protocol Properties (Generic and Specific). These Protocol Properties can still be modified by the application. Basically, Protocol Properties are the only thing that the application can still set, everything else it can only query at this point. Finally, the application can query properties of the path. Not sure how well we have to specify them, probably this part will be implementation-specific just like, e.g., the Interface Instance or Type. Does everyone agree, or is there anything to discuss here? For me this is not a big change, it's just putting what was already there in terms of . The only thing I'm wondering if whether the Protocol Selection Properties of the finally chosen protocols are still "Selection Properties", as the selection is already done. That's why they were called "Transport Features" before, but now I want less terms and they're almost the same thing...
OK - so I commented on these things before - but in these things I think I agree. I'd personally prefer to keep the features as attributes implemented in the transport (etc) and the properties as the things requested. Indeed this begs the question about what to call the resulting transport when you selected properties and now get a transport. I think still calling them (for consistency) properties seems really good to me. :-).
As QUIC has unidirectional streams, we might want to be able to accommodate them in our transport API. Do we want to handle them fundamentally different from a half-closed connection? This should match the decisions made in .
Side note: we don't support half-closed connections to begin with. As for unidirectional streams, is this the final decision in QUIC's design, or might it change? I'd suggest postponing this until we can be sure.
NAME regarding half-closed connections: we don't have any top-level state in the connection for being half-closed. However, being able to express that a message is final in a given direction when sending allows the same functionality. As I've discussed before, we absolutely need the ability to express sending a TCP FIN, but I whole-heartedly agree it doesn't need to be a top level concept. I think this is somewhat different from half-closing, since a unidirectional stream never allows sending on one end. You could imagine that it is a connection object that returns an error if you ever try to send on it, etc. If you wanted it to look like half-closing, you could open a connection (stream), and then mark it complete in a direction immediately? But that's a bit odd.
All ok, I agree with your suggestion to fin-mark a message
I agreee with NAME regarding half-closed. My suggestion is to explicitly state that connections can be uni- or bi-directional and to add a method to query that (cansend / canrecv). This aligns nicely with final messages after which can_send becomes false. I can do a PR for that.
Thanks! I like this
Still missing unidirectional stream for multicast. – please continue discussion of multicast aspects in The connection properties (can send/receive data) are still a little under-specified - please discuss in details in – we should fix this together with .
We need unidirectional connections. You know, for multicast. And ICN. And unidirectional streaming protocols.
This is a dup of , which has more discussion
I am re-opening this ticket as only dealt with unidirectional streams in unicast context (added with PR ): Current status: The current model for unidirectional streams supports half-closed connections and QUIC style unidirectional streams so far. We don't know yet whether it is sufficient to have canread/canwrite property on the connection level to model unidirectional connections We don't have pre-connection methods to create multicast sessions yet
there's a selection property for this, tagged discuss, which we haven't discussed yet. I'm moving this down to the appendix as part of my restructuring in my work on
If help is still wanted here, I'm maybe up for an attempt or 2 at adding multicast support. Has there been some prior discussion I can review about rough ideas on pre-connection methods for multicast sessions, or should I just try making something up?

ok, with the caveat that we need to keep multicast as an open issueLGTM About the multicast concern: I agree that it would be good if this could function for multicast reception too, and I couldn't tell if this is suitable for multicast. I suspect that multicast support is much harder anyway - see RFC 3678... so for now, this is a good step forward IMO, and let's discuss how this relates to multicast when we see a concrete text proposal that addresses it.


This is the second step towards applying our new Properties Classification. It moves up the Connection Properties and annotates most places with stuff to do.
As discussed in
Thanks for moving this forward! I only found a few nits. Also, I have a question about whether two of these Properties should maybe be changed to be Selection Properties, but that's more of a discussion and not something that needs to be done in this URL modulo NAME commentsI carefully checked it, but I find it very hard to be sure that this is 100% ok (as is usually the case IMO with diff's about text blocks being moved around). My suggestion is to land it and see.


Message Context Parameters/Properties are now Message Properties, and associated terminology cleanup. Addresses part of the resolution to .
Looks good, thanks for the cleanup!LGTM


Thanks for these fixes! I did a bit of reworking of the abstract, but it keeps the main part of the changes you made.
Regarding the sentence you removed, that was an adaption of an existing sentence, which I believe aimed to mention the words implementation and API to implicitly refere to the other docs (which was probably too subtle for anybody to notice anyway). However, the other change I made to that sentence was to mention one of the benefits / deployment incentives for taps (dynamic protocol and path selection). I wasn't sure about the wording here, but I think it would be good to (re-)add a sentence to the abstract saying why taps is so great and that everybody should deploy it.
Yes, the original second sentence was aimed at mentioning that there were two other documents, API and Implementation, that would be based on this one. I liked how in your revision, it did try to make things clearer than just saying "flexible transport networking services", but I felt that the specific example of dynamic path and protocol selection was a bit too specific for that one line. (I was chatting with NAME about this as I was editing, btw). My thought was that the list of features that you added in right after that, which go through the top-level bullet points of the advantages, do a more comprehensive job of summarizing the benefit.


Several small updates, mostly to clean things up a bit, and align property names and such.
NAME - I think you're wrong: my understanding (and also the way the text is written - otherwise we'd have to change it in the API draft) was that Connection Properties can (and should) also be set on Preconnections, to give as much input as possible for Connection establishment. In the API draft, the Selection Properties in Section 5.2 can only be used on Preconnections: you can't ask for congestion control after establishment. You can't say that you'd rather have multistreaming support after establishment. But that doesn't exclude Connection properties from being usable earlier. Required minimum checksum coverage could influence choosing UDP-Lite vs. UDP. Bounds on Send or Receive Rate could influence the interface choice. The capacity profile could influence various things (and no, there isn't a matching Selection Property now - there is a per-Message override). So... I think that either you're wrong, or I'm missing something and the API draft heavily lags behind a decision that was made. I hope not!
NAME The API drafts says: I interpret this as only the Selection Properties influence selection and the Connection properties are used to further configure the connections according to user preferences. Otherwise what is the difference between them?
If you interpret it as you do, I think the text isn't clear enough (indeed it isn't super clear - in particular, "eventually established" is misleading IMO). Though, note that it says that Connection properties can also be used later. Regarding the difference between them: Selection Properties only make sense for Preconnections.
NAME - sorry lost this thread. But I saw that you have started cleaning up some Connection properties that had mixed Connection and Selection properties up in the API draft. Does it mean that we are now in synch and agree that only Selection properties are used in the selection and Connection properties are used for configuration (both during establisment and also after the connection setup).
NAME the other PR means nothing about what I said here: it only tried to address issue which is about a wording inconsistency in a specific property. You say that Connection properties can also be used during establishment - if you agree with allowing this, then why not allow to configure them as early as possible and allow them to influence selection? What's bad about having more information available that can guide the selection process?
NAME the problem is that the Connection properties are not expressed in a way suitable for selection. If they should also be used for Selection they would have to be reworked to be of type Preference. Otherwise I do not know what they mean for selection. What does it mean if I pick say a particular connection group transmission scheduler? Is it required or preferred? Most connection properties also do not look very useful to me for selection so in case there are some we need I think it would be simpler to just define a matching selection property.
NAME we're only discussing whether it's allowed to use them as additional input or not. I agree, some (like the scheduler choice) don't seem useful, but some do... generally we may not have good guidance on how they should be used by an implementation, but giving the system freedom to use as many inputs as possible is probably a good thing. Regarding the type, require/prefer//ignore/avoid/prohibit obviously works fine for a "wishlist" of boolean character: having something, or not having it. The selection properties are generally like that - except "Interface Instance or Type" and "Provisioning Domain Instance or Type" which are, well... wishlist-collections. Consider the capacity profile. Surely configuring this early can be useful input - e.g. if someone picks low latency / interactive, you may even want to put this traffic on a specific network interface! But if you'd want to make a copy of the capacity profile as a "proper" selection property, how would you do it? Apply require/prefer/ignore/avoid/prohibit for each of its values? Then I could say "prefer" low latency/non-interactive but "avoid" low latency/non-interactive, which would surely be odd. Apply the qualifier to the whole thing? Then it's about, e.g. "avoiding" to use a capacity profile at all. That's odd as well. I think it's in the nature of the current selection properties that they merely say "I want to be able to do this, and this is how important it is to me", which is different from configuring something. That's also why, as written, they simply wouldn't work for later connection configuration. Connection properties really are about configuring something, but I still say that being able to use them as early input is good. I agree that the implementation isn't super clear as they don't come with a require/prefer/... etc. qualifier, but then we just offer some freedom to implementers.
NAME I just pushed a commit that updates the selection and connection properties text, removing "connection" almost everywhere, but adding a disclaimer in the beginning that says that connection properties should be configured as early as possible because they can influence decisions made by the implementation, and "In the remainder of this document, we only refer to Selection Properties because they are the more typical case." Frankly I'm not sure this is better... but it does remove most of the "selection and connection" double mentions that you pointed out.
Fixed up branch building and resolved merge conflicts. I'll merge this in later today before the posting deadline.
Cool thanks! Sorry if I messed something up
As discussed in
Thank you for updating this – looks good! just needs s/send-params/msg-properties/ and merging


Closed
I just read Arch and have the following NiTs which I think we should address: /Listeners by default register with multiple paths/ ought this to be: /Listeners, by default, register with multiple paths/ /A Framer is a data translation layer that can be added to a Connection to define how application-level Messages are transmitted over a transport protocol. / I think this should be (because I don’t think they are protocol specific) /A Framer is a data translation layer that can be added to a Connection to define how application-level Messages are transmitted over the transport service. / /to retransmit in the event of packet loss,/ better to avoid event here :-) /to retransmit when there is packet loss,/
I am just gonna add a few more nits here: The Listener object in figure 4 is a bit awkward, it appears as if the Listen() call appears out of thin air while per the API it gets called on the preconnection object like Initiate() and Rendezvous(). In section 4.1.1, there are 5 references to section 4.1.2, is it really necessary to have that many? Point 2 in section 4.2.3 only talks about required transport services, the same should probably be said about prohibited services, i.e. "both protocol stacks MUST NOT offer transport services prohibited by the application"
Let me add to this: Section 1: print send() and recv() in fixed-with font [HTML/XML issue only] Section 2 - Figure 1: socket() [with the brackets] looks odd in the figure title
NAME I fear framers may be specific to the underlaying protocol, e.g., for SIP slicing messages out of TCP stream whereby doing nothing for SCTP and UDP
In relation to frames I think I only asked to add /service/ instead of protocol... I don't understand how Phil's comment reacts to that
NAME I think protocol is correct here and would not change the sentence.
OK - so I think you are saying the framer depends only on the specific next protocol, rather than upon the transport service that was offered? - It is still not clear to me that this is correct.
NAME that is a fairly good question… The behaviour of the framer defiantly depends on the transport service provided (which may be a subset or superset of the transport service requested). A framer needs to support all possible outcomes of the transport services requested It may optimize based on the protocols it actually finds, e.g., turn into a no-op with the transport preserves message boundaries. Whether the next protocol is a sufficient representation of this is unclear to me. Maybe not, if one thinks of supporting arbitrarily large messages and UDP over IPv6 with and without fragmentation…
LGTM, thanks!


Just some points about the new text on framers in the API draft: In 10.2 it is specified in which order multiple framers run on incoming and outgoing messages, it is however never specified what the order looks like on Start and Stop events. In 10.6, "MessageFramer.DeliverAndAdvanceReceiveCursor" appears to be missing the "Data" argument that is present in "MessageFramer.Deliver". Also in 10.6, in the second paragraph it says: When it is probably meant to say While the framer implementation is able to notify the Connection that it has finished handling a Start event (by calling MessageFramer.MakeConnectionReady), there is no such way for Stop events.
As notes for the responses to these points: If there are multiple framers, the "lowest" one (the one closest to the transport) would receive start first, and the next would get the start event when the lower one is ready. Similar for stop. Good calls on the missing data argument and the typo re application vs framer. I was thinking to use the failConnection call to complete a stop, potentially with no error?


Addresses
The architecture draft uses UDP checksums as an example of a protocol-specific property in Section 3.2 and 4.1.2. I find this a bit confusing because in the Interface draft, checksum coverage is a Generic Connection Property (Section 11.1.2). Should we maybe use a different example of a protocol-specific property, such as the TCP User Timeout we have in the Interface draft?
Sure, we can switch the example around
Thanks, LGTM!Thanks – works


Took a stab at normative language for the rules to process Selection Properties, as discussed in . As far as I remember, we always said that outcomes have to be deterministic/consistent, so I made everything that matters a MUST. I guess our example illustrates that even given the same Selection Properties, the outcome might change depending on the situation, e.g., your path might still be Wifi but it might or might not support your preferred protocol. So, does this really have to be all MUST, or is there a SHOULD in there? Discuss. :) Also, if anyone can name a good reason why the outcome has to be deterministic, please suggest text - maybe something like "avoid surprises for the application", but this didn't really sound right, so I left it out for now.
In section 5.2. in the API doc: "Internally, the transport system will first exclude all protocols and paths that match a Prohibit, then exclude all protocols and paths that do not match a Require, then sort candidates according to Preferred properties, and then use Avoided properties as a tiebreaker. Selection Properties that select paths take preference over those that select protocols. For example, if an application indicates a preference for a specific path by specifying an interface, but also a preference for a protocol not available on this path, the transport system will try the path first, ignoring the preference." This sounds rather like an example algorithm that should maybe go into the implementation doc instead? Or is there any part of this that we need to specify normatively in order to ensure that all systems implement the same?
This goes back to issue on "conflicting policies". At the time, we decided that there must be a deterministic outcome, therefore, text was added to both the API and the Implementation document, see . If we keep it that way, perhaps the text here should point out that the goal is to resolve conflicting policies, so we achieve a deterministic outcome.
Ah okay, thanks for the background info; missed that. I think we need to use normative language here then. Maybe also a separate subsection makes this more prominent.
This reads to me like the rules for processing polices rather than an example algorithm. I think it is part of the API... Normative language would be good and a stronger opening that explains why the rules exist.
I don't think a separate subsection would make this a better read - this fits fine as it stands, as an intro to the transport property specification IMO. Regarding normative language, that's a separate discussion. So, can we close this issue?
Agree that we don't need a separate subsection. I can try to tweak the beginning of this part a bit to explain why the rules exist as NAME suggested. Regarding normative language - we definitely have it in the API draft, right? So why not have the discussion on it here as well? if we're serious about deterministic outcomes, I guess the whole thing needs to be a MUST, right?
Sorry, true, that normative language debate was about the arch draft, and we definitely have it in the API draft. So yes, let's discuss it here. I also think a MUST would make sense.
So the order of filtering out require/prohibit is not important (outcome is the same) - making the order a MUST does not make sense. Also, first sorting by preference and then filtering has the same outcome as doing it the other way around. the only thing we need to say is that filtering MUST happen and sorting SHOULD happen.
ok - I have to admit I didn't think this through, I just agree that we want a deterministic outcome from this.
Why did we require a deterministic behavior in first place? Does this prevent TAPS implementations from running experiments like Apple did with TCP SACK and ECN? Does this prevent probabilistic path probing?
This was discussed in the TAPS BoF? ... I thought it was about developers being assured that X will happen if they choose Y and not finding the ground shifting as they use the interface.
Interim: Prefers and avoids allow the implementation to decide the ordering and weighting; require and prohibit are the only hard requirements. Applications should be able to cope with any possibility that isn't prohibited. Likely should add implementation text to say how preferences weight racing.
During the interim discussion, we found that was too strict and limiting, so we opted for another approach that moves the selection algorithm to the Implementation draft. I'm preparing another PR implementing these changes.
The text that ended up in the implementation draft is not fully consistent with the racing text, opening a new issue for this and closing this issue.
LGTM, merging this. thanks, Theresa!This reads good to me. Regarding the request for text, I don't have any ideas... but I think this could be fine as it stands? Needs more eyes though.


This ended up in more text that I thought. I'm not certain if we need all of it. Further I'm unsure about the concrete wording in many cases, so any comments are more than welcome!
This is supposed to address issue
I do mentioned name resolution, however, leakage though DNS might rather be a DNS problem and not necessarily an issue that needs to be considered in taps. Sharing cache state is a good point. I vaguely remember that we already address that somewhere else (could also be in the implementation draft as this is rather an implementation issue). But maybe we can add a pointer...?
Leakage though DNS may need to be addressed by TAPS, as with the current sockets API, the application has complete control about when DNS resolution is done (and, at least as long as things like DoH/DoT are predominantly implemented by nonsystem resolver libraries, how). TAPS as envisioned gives the system more control over resolution.
(IIRC the architecture does explicitly address and mention the shared cache issue; adding a line to refer to that in privacy considerations might be nice but isn't IMO necessary.)
Gorry, I addressed all your nits/comments.
I think it is a good start, but I miss two aspects: Information leakage through DNS, which can also make addresses on different paths linkable Information leakage within an application though shared caches, e.g., within a browser. I agree this is a good starting point; note that I took the liberty to directly fix some nits (such as capitalizing TAPS) and typos; these never change the intended meaning.


Would be nice if someone has time to do some ascii art :-)
Wouldn't that be fig. 4 in the architecture draft? Except that, looking at this now, I see inconsistencies! E.g., I don't see "RendezvousDone" in the figure. Also, instead of ConnectionReceived, it says "Connection Received" - rather, this should probably marked as an event by writing it as "ConnectionReceived<>". The figure is also missing InitiateWithSend() ... probably it would be good to make this figure both complete and consistent with section 13, "Connection State and Ordering of Operations and Events", of the API draft? I don't think very much is missing there.
I think it would be nice to have a real state diagram here, probably with states like "closed", "open" and "listen" and the events that create a transition between these state.
I'm not sure that Figure 4 in the Arch diagram serves the purpose that Mirja's trying to address with the state diagram in API. I also don't think we need FIgure 4 to align with the specific naming and CamelCasingConventions in API.
Thanks! I guess you could add a loop for sent<> and received<> in established state but maybe that also too much.Works for me


This proposal consolidates normative language into the section on design principles (now requirements), using the suggestions in .
Please review the latest updates!
Early art-art review of draft-ietf-taps-architecture My first observation and request is that the group rethink the separation of the architecture and interface documents. It is a warning-sign that the interface document is a normative reference for the architecture document. As the documents are currently constructed, a new reader cannot grasp the architecture without inferring some of it through reading the interface. Consider reformulating the architecture document as an overview - introduce principles, concepts, and terminology, but don't attempt to be normative. (There are signs that this may already be, or have been, a path the document was being pushed towards). If nothing else, rename it, and acknowledge that the actual architecture is specified using the interface definition. Note that the first sentence of the Overview in the interface document also highlights the current structural problem.
I don't entirely agree with this ART-ART comment: I think I might well agree with moving more of the description to this Arch document, but as discussed before, I also think it useful here to set the actual requirements for the architecture, and to define the architecture. It may need to be much clearer that this architecture consists of more the API.
For completeness, this is the first statement of the interface document review: This document reads fairly well. I have some structural concerns (see the early review of the architecture document). In particular I think this document contains the actual definition of the architecture and some of that is inferential.
Discussion at September 2021 interim: pass through this document with a view toward making it a high level introduction, not an architecture definition: "An Introduction to the Transport Services Architecture". remove normative language from this document (moving it to API, where necessary). move details not useful in a high level intro to API.
I disagree with this conclusion at first pass—we did a specific analysis and pass on this over a year ago, with PR . The normative language and the standards-track aspect was very intentional and something the WG had consensus on. The point is that the API is a specific incarnation of the TAPS architecture, but there are overarching architectural properties that go beyond a specific abstract API.
I completely agree with Tommy here, and recommend we fix this in . We should also of course check for any stray text or missing concepts. I think it would also be useful to somehow better explain that in figure 3 the API function is ONE part of the system.
See linked PR first please to see if this resolves this?
A PR was merged that addressed important text issues with separation of requirements and overview, and should more clearly set out that the requirements are for the system. The question of whether the overall TAPS systems requirements belong in this ID; the API; or a separate RFC needs some discussion based on the review, before this can be closed.
Many good text improvements in the PR NAME but I noticed two smaller updates in the PR that I think were incorrect. Fixed in
The review comment showed were many places where the wording of architecture can be improved. There are some high level principles that I think should remain in the architecture document. The decision in today's interim is to review the architecture to get this more into a shape that is valuable as a reference, and then check the updated draft.
I now suggest the issues are addressed and we close this issue without moving text between documents
I agree with that; closing.
Overall review: I think we need to keep the normative language for the key features that differentiate the API. I'd really like to use REQUIRED, RECOMMENDED, rather MUST and SHOULD - because these are not protocol operation requirements, but are implementation requirements to conform. This includes requirements based on Minset - where the WG dervied this API: This is after all why the WG worked on Minset as the basis of the API. I suggest we group all the requirements on architecture as short one or two sentence blocks (preferably as a bullet or definition list) collected into a single subsection, so they are easy to find and read and we say that these are the key architectural requirements. We insert this new requirements text either their own section, or in section 1.3 (although I am open to other places where we can call-out the architectural requirements). The following are key requirements, that define the TAPS architecture: It is RECOMMENDED that functionality that is common across multiple transport protocols is made accessible through a unified set of TAPS interface calls. As a baseline, a Transport Services system interface is REQUIRED to allow access to the minimal set of features offered by transport protocols {{?I-D.ietf-taps-minset}}. A Transport Services system automates the selection of protocols and features and network, based on a set of Properties supplied through the TAPS interface. It is RECOMMENDED that a Transport Services system offers Properties that are common to multiple transport protocols. Specifying common Properties enables a system to appropriately select between protocols that offer equivalent features. This design permits evolution of the transport protocols and functions without affecting the programs using the system. Specifying common Properties enables a Transport Services system to appropriately select an appropriate network interface. It is RECOMMENDED that a system offers Properties that are common to a variety network layer interfaces and paths. This design permits racing of different network paths without affecting the programs using the system. Applications using the Transport Services system interface are REQUIRED to be robust to the automated selection provided by the system, including the possibility to express requirements and preferences that constrain the choices that can be made by the system. If two different Protocol Stacks can be safely swapped, or raced in parallel (see Section 4.2.2), by the Transport Services system then they are considered to be "equivalent". Equivalent Protocol Stacks need to meet the requirements for racing specified in section XX. Applications using a Transport Services system MAY explicitly require or prevent the use of specific transport features (and transport protocols) and XXXinterfaces using the "REQUIRED" and "PROHIBIT" preferences. This allows an application to constrain the set of protocols and features that will be selected for the transport service. It is RECOMMENDED that the default usage by applications does not necessarily restrict this selection, so that the system can make informed choices (e.g., based on the availability of interfaces or set of transport protocols available for the specified remote endpoint). When a Transport Services system races between two different Protocol Stacks, both SHOULD use the same security protocols and options. However, a Transport Services system MAY race different security protocols, e.g., if the application explicitly specifies that it considers them equivalent. Transport Services systems MUST NOT automatically fall back from secure protocols to insecure protocols, or to weaker versions of secure protocols. A Transport Services system MAY allow applications to specify that fallback to a specific other version of a protocol is allowed. It is RECOMMENDED that in normal use, a Transport Services system is designed so that all selection decisions result in consistent choices for the same network conditions when they have the same set of preferences. This is intended to provide predictable outcomes to the application using the transport service. I would then suggest to use lower case within other parts of the document - except were pointed to from here, (we could refer to the requirements subsection, if we thought necessary).
for discussion at the interim, but IMO we could start the interim with a PR based on these suggestions and work from there.
I suspect that we need more protocol-specific properties to be able to talk to various kinds of native protocol peers (other than TCP and UDP: these should already work fine with what we have, I believe). A concrete example: we're hiding stream numbers. In NEAT, we needed to make them configurable though, because we wanted a NEAT client to be able to talk to a native SCTP server, where an application may expect certain type of data to arrive via a certain stream number. I suspect that we may need something similar for a QUIC peer. Should we address this? (I'd say yes) How .. is it a protocol-specific (per-Message) property? (I'd say yes... and perhaps a per-Connection property too, with text explaining that the per-Message one automatically becomes the Connection default unless otherwise specified) What other such things are we missing... what other expectations may e.g. a native QUIC or SCTP application have that we just cannot fulfil with our current system?


In the API doc sec 5 says. "Message Framers (see Section 10), if required, should be added to the Preconnection during pre-establishment." Should this be SHOULD or even MUST? If SHOULD is correct I think we need to say slightly more about when it makes sense to add a framer later.
I think it should be "can be added..."; not normative, just descriptive of fact.
Can we start with the "If' perhaps... "If messages frames are to be used, they MUST be added to the Preconnection during pre-establishment."?


I actually really like: "Simultaneous" and "Staggered" as descriptive terms that avoid the question of "delaying".
Opening this to follow up on a comment by NAME on that is now closed. "Simultaneous" and "Staggered" would be better names than "Immediate" and "Delayed". Just from these terms, I would normally think that "Immediate" means to start racing right away, whereas "Delayed" means that nothing would happen for some time. So these two terms are slightly misleading in my opinion.)
I agree this would be an improvement.
In Section 4.2.1., I didn't really understand what "immediate" means from the text. I think it just needs one sentence.
lgtm (side note - nothing blocking, this is just an idea: to me, "Simultaneous" and "Staggered" would be better names than "Immediate" and "Delayed". Just from these terms, I would normally think that "Immediate" means to start racing right away, whereas "Delayed" means that nothing would happen for some time. So these two terms are slightly misleading in my opinion.)Looks good, thanks! I will close this and open a separate issue for the naming.


NAME 7df2d9c should address this
To we need multiple local and remote candidates. Should this be represented as either: 1) taking two lists of objects; or 2) as two objects each with multiple addresses, ports, etc. I suggest option 1, since this makes it easier to resolve a to multiple preconnections, for NAT binding discovery. Relates to but with also different types of endpoint.
I like option 1 because it contains complexity.
See this email thread: URL Copy + pasting a relevant part from an email from NAME To me at least, it wasn't clear from the text that rendezvous() would kick off ICE-style probing. It's kind of obvious (when else would it happen?), but I think it would be good for the draft to spell this out.
modulo the do-we-want-to-bind-tightly-to-ICE question... Dear all, I know way too little about rendezvous mechanisms, so I may get this completely wrong - please bear with me! It may be that I describe an issue that’s a non-issue. But, if I do understand this right, maybe that’s something which needs fixing? Ok, here it goes… Our current Rendezvous text in the API draft says the following: "The Rendezvous() Action causes the Preconnection to listen on the Local Endpoint for an incoming Connection from the Remote Endpoint, while simultaneously trying to establish a Connection from the Local Endpoint to the Remote Endpoint. This corresponds to a TCP simultaneous open, for example.” Now, if an application chooses unreliable data transmission, it may get SCTP with partial reliability, or it may (in the future) get QUIC with datagrams, or whatever… and then, the protocol may be able to do what’s needed (something similar to a simultaneous open). However, in case of UDP: Listening is quite okay, “simultaneously trying to establish a Connection” really means nothing - no packet will be sent. I believe that applications doing this over UDP will implement their own “SYN”-type signal for this - but then it’s up to the application, which, over TAPS, would need to: create a passive connection, send a Message from that same connection, i.e. using the same local address and port (I believe?) - which is a matter of configuring the local endpoint, and doable if the underlying transport system can pull it off. I think there’s nothing in our API preventing an application from doing this. If all of the above is correct, the problem that I see is that an application may ask for unreliable transmission, and then have one case in which calling “Rendezvous” is the right thing to do, and another case in which it is not - so it would need to be able to distinguish between these two cases: is it a protocol with Rendezvous-capability, or do I have to do it myself. Maybe, in case of UDP, the call to Rendezvous should just fail, and our text should say that Rendezvous failing would inform the application that it would have to implement it by itself, by issuing Listen and then sending a Message to the peer. Thoughts? Am I completely off track, or is this something that needs a text fix? Cheers, Michael


(well, the priority framework is still confusing, this PR makes it clear that that is intentional, and why)
Martin (I think) said that he'd check this offline after the Sep 11 interim.
At 8.1.2 you say lower values have higher priorities for connPrio. But later, at msgPrio, higher values have higher priority, and in the examples where you speak of the interaction of connPrio and msgPrio, you imply that higher means higher for both properties. Please check for consistency and clarify.
Lower values having higher priorities for connPrio is a remnant that we must have missed when we flipped this, e.g. PR which I don't know... too many iterations have mangled the text.
Aha, now I see what happened: following some more PRs, also , in PR , NAME re-introduced some text saying that sends on Connections with lower-priority values will be prioritized over sends on Connections with a higher-priority value. This reverses the logic for some part of the text - I suspect that NAME (very understandably!) forgot what happened and this created this mix. We have been editing these documents for way too long!
I'm not sure the priority framework really fits together. Sec 6.4 says we 'should' do something that sounds like WFQ, but 7.1.5 allows the application to decide what happens. Meanwhile message priority suggests a strict priority-based scheduler. Meanwhile, in sec 8.1.3.2 "Note that this property is not a per-message override of the connection Priority - see Section 7.1.3. Both Priority properties may interact, but can be used independently and be realized by different mechanisms." At the risk of overindexing on a partly-baked draft in httpbis, how would I implement strict priority of HTTP/3 streams over a taps API that implements QUIC? Would I need to request strict priority in both connection priority and message priority? Or I guess connection priority would be sufficient?
We seem comfortable with the mechanisms in the API, but we should have more text connecting the various sections explaining priority.
My comment concerned "7.1.3. Priority (Connection)", my thought was this just influences the sender-side to control queuing within a connection group. If so, if it worth calling this property "Group Priority" and explain this is sender-side?
I find the priority framework perhaps overspecified at the connection level and underspecified at the message level, with the interaction between the two left explicitly unstated. I would almost rather see the priority type and meaning left completely implementation-specific; otherwise, my suspicion is that someone will write an application using numeric priorities, declare victory when it works fine on one platform, and not understand that the behavior is not universal. In general, I think if TAPS is going to produce a consistent, easily-understandable, and portable API, it needs to stick to easily-understood abstractions. Message-level priority is sort of like the TCP URG mechanism (though better specified and more useful, for sure), but I'm thinking urgent messages sent explicitly out-of-band (e.g., new Connection) with implementation-specific QoS will produce more consistent, predictable, and understandable outcomes in the pessimal case.
Ok for me (and thanks for doing this!)Some nits, but I believe this addresses my concerns in the issue.


This provides more detail on how to implement Clone with TCP, and clarifies that SCTP's "Initiate" should not yield a stream of an already existing SCTP Association in case "activeReadBeforeSend" is Preferred or Required.
Currently the "Clone" part of, e.g., TCP (section 10.1) only says: "Calling Clone on a TCP Connection creates a new Connection with equivalent parameters. The two Connections are otherwise independent." This text should reflect that equivalent behavior must be maintained upon later Connection configuration calls. And, there should be more text in general on cloning (see issue ).
This text must refer to a new "do I want a Connection that supports initiate followed by listen?" property. This property will be added to address - so I'll address this issue when the PR addressing has landed.
Added a question for NAME


Now I feel like I'm shaving the yak. This is more than what we talked about in the interim, but it felt like the following sentence required additional clarity. Let me know if I misunderstood the intended meaning, in which case I will rephrase.


This changes the use of the term "entanglement" as discussed in , and moves some text around in this section for better clarity and less repetition. Also, addressing , it introduces a framer as an optional parameter to the Clone call.
NAME I incorporated all of your suggestions, thanks!
This is not fully clear from section 5.4., however, if connections in a group are each QUIC streams, I think it should be possible to use different framers on each stream. So it might be okay that a framers is copied when clone is called but changing a framer on one connection should not change the framer on the other connection in a group...?
Also an related editorial comment (that could be addressed in the same PR eventually): this section on groups only at the very end has a paragraph starting with "Note that calling Clone() can result in on-the-wire signaling..." which kind of explains that groups are supposed to be used for multistream protocol but can also be mapped to separate connection if the protocol doesn't not support multiplexing of stream on one connection. I think this information should be added at the very beginning of this section.
Actually, now looking at the framer section, I guess you would rather like to already set the new framer before you clone a connection...?
I agree with everything you say here; about your last suggestion, maybe a framer should be an optional argument of the clone call?
I don’t actually understand the difference between a connection in a Connection Group and an “entangled connection” ? Both terms are used in API, but in different ways (possibly by different authors) - do we really really need the word “entangled”?? if we do we need, can we define and explain this.
I think that having the word is good as it indicates a deeper relationship than mere "grouping". Just based on terms, I'd (correctly) imagine that I can define priorities between members of a group, and that they somehow belong together... as in: Close with a common call, etc. ... but I don't think that I would expect, just from this term, that changing a property on one of them affects all the others. Hence the phrasing: "The connections within a group are 'entangled' with each other". However, I agree that in most other places, we should talk about "groups" instead. I can try to fix this.
Sounds good - we should add that explanation upfront in the definitions then also?
[Yes, I think so.] EDIT: no, now I don't think so anymore. (see )
I have addressed this in PR . Note that the text interchangeably talked about both Connections and Connection Properties as being "entangled". I have now written this such that the term "entanglement" applies to Connection Properties only, whereas Connections in a group are, well, members of the same Connection Group.
Looks good to me.


The text says What does disabling mean? Does this mean mean that it is valid to set this property to "Prohibit" but still set "safelyReplayable" to "Require" on the PreConnection. Should that not be an error? This is also related to . The use of safelyReplayable for both reliable and unreliable services makes the text a bit confusing. Would it not be sufficient to state that if you do not Require Reliable Data Transfer you have to be prepared to deal with duplication?
We should remove this sentence in 4.2.5:


For other parameters you have to pass in an empty object if you do not want to set anything but not for SecurityParameters. Unclear why the parameters are treated differently?
Make security parameters non-optional Remove "Transport Properties MUST always be specified while security parameters are OPTIONAL." Define that there is an implementation-specific security parameter to disable security
LGTM, thanks!


This tries to write something small but specific about Network Policies
LGTM
In Sec 2: Should Section 3.2 also talk about network policies, e.g. a certain network supports a DSCP that the application desires?
As discussed in Feb 2021 interim, this seems like a rabbithole. Gorry will figure out whether it is and write language if not, or close if so.
I'd like to add a couple of lines of text here to bullet 2 that discusses how you might learn that an attached network supports certain features - e.g. through a PvD, routing advertisement, or local discovery method, and suggest that this could enable a system to know which paths have specific characteristics.
I propose this as a starter: When additional information (such as Provisioning Domain (PvD) information {{RFC7556}}) is provided about the networks over which an endpoint can operate, this can inform the selection between alternate network paths. Such information includes network segment PMTU, set of supported DSCPs, expected usage, cost, etc. The usage of this information by the Transport Services API is generally independent of the specific mechanism/protocol that was used to receive the information (e.g. zero-conf, DHCP, or IPv6 RA). Thoughts... ?
Tried to create a PR to add some policy to the path selection
I think the policy is separate from the API.LGTM


Applied requested changes
In 3.2 it is clear that you can set Message Properties on Connections and Preconnections, but the rest of the document is not very clear about this. In 4.2 for instance the text only talks about setting Selection Properties and Connection Properties on PreConnections. In 7.1.3 the text says "Connection Properties describe the default behavior for all Messages on a Connection". But I think that Message Properties set on a Connection or PreConnection may also describe the default behavior so this relationship should be clarified. Also, the Message Property defaults given for Properties that do not depend on other Properties sets the defaults for all Messages in a Connection I assume? I also assume that a Message Property set on a Message Context overrides a Message Property set on a PreConnection or Connection, but this is also not specified. In 5.4. Connection Groups, I assume that setting Message Properties on a Connection applies to all Connections in the Connection Group? This is not clear.
In general, I agree. If you set a message property dynamically on a connection, though, I don't think it would affect the group—I could have one stream in a multiplexed protocol have separate behavior, such as priority.
+1 to tfpauly. But this is already written in 5.4, maybe you missed it (Anna)? "Message Properties are also not entangled. For example, changing Lifetime (see Section 7.1.3.1) of a Message will only affect a single Message on a single Connection."
I am fine with an update of a Message Property to a connection applying only to that connection. Priority is not a good example as it is already a special case, but my main issue was that we need to be more clear about the behavior in the specification. NAME the issue is about setting a Message Property on a Connection. The text you refer to talk about setting Message Properties on a message. You could interpret the first part as a general statement, but the way the text is structured it is easy to think it refers to setting Message Properties on a message so I think we should be explicit also about setting Message Properties on a Connection. Could be a second example.
So after looking at the text again, I think the example talking about setting Message Properties on a message should be removed, see comment in the pull request.
Thanks!!


Explain the different use-cases on specifying interfaces to
There's something obvious I'm not seeing: if I want to specify a multicast or link-local address for an endpoint, why not simply specify that address? What's the use of adding the interface into it? Some addresses will inherently limit interfaces; I thought interface preference was about preferring wifi to 4G, etc.
Like, in section 6.1.1:
Multicast groups can be scoped (see RFC7346), thus, may only be available on certain interfaces or even have different meanings based on the interface. So for Interface- and Link-Local scope, specifying the interface is strictly required. For others use cases (like coping with non-unique use of RFC1918 addresses) this may also be helpful.
Why does the API have both the ability to say LocalSpecifier.WithInterface and to set interface requirements on TransportProperties? Why not just keep the latter and remove the former?
There's history here: LocalSpecifier.WithInterface predates explicitly setting the interface name in interface requirements. There might be reasons to keep this: LocalSpecifier is more likely to show up in application UI than in transport properties. Any text proposed to keep both would need to have an unambiguous way to resolve conflicts between the two. This might be a source of hard-to-debug errors.
To discuss next interim


Sections 4.1 and 4.1.3 introduce to configure an Endpoint Object. The address supplied in examples appears to be an address literal. However, Section 1.1 does not define any types explicitly associated with an internet address -- or could work, but I don't think that's the intent for these APIs, either. Section 3.2.2 does not extend this. It's unclear whether this literal is intended to support IPv6 scoped addresses as described in [RFC4007]. While this RFC describes a textual representation (possibly not in line with the goals of the API), [RFC6874] describes how to represent zone indexes in an address literal. If zone indexes are not intended to be supported, this could be clarified by, when "IPv4 and IPv6 address" is mentioned in section 4.1, also citing [RFC791] and [RFC4007] with the respective address family. If zone indexes are intended to be supported, ambiguity exists around the use of and on a Local Endpoint, where the supplied address has a zone index that does not match the supplied interface. This could be clarified by: Defining type names that represent IPv4 and IPv6 addresses in section 1.1. Specifying the properties of an address literal supported by that type (referencing [RFC791] for IPv4 and both [RFC4007] and [RFC6874] for IPv6). Clarifying in section 4.1 that "IPv4 or IPv6 address" are of the respective type described in section 1.1 when used in or . Clarifying the behavior of zone index and interface choices on a Local Endpoint in a new section 4.1.4. This may tie in to to some extent in that this is unlikely to be discoverable until is called. [RFC791]: URL [RFC4007]: URL [RFC6874]: URL
(note this is in §6.1/6.1.3 post-)
Mention that the %en0 addition to the IPv6 address is equivalent to explicitly setting WithInterface. Also mention that these are implied to be system types for addresses
Looks sane