This is a initial PR to get some basic definition of subscription hints. I would like to get us to an agreement for the basic proposal and then discuss further on error codes, edge cases and impacts/interactions with other messages (Sub Fin, RST, UPD)
Can you mark this as so it shows as having a PR attached (also possibly 111 and 260)? Also recommend closing as this replaces it.
I feel proposal is more ambiguous though. I agree we can make the proposed ending in this PR to make it easier to encode/decode. I have to get higher level idea agreed upon There are subscription hints. 2 hints are defined in this PR a) StartPoint b) Interval There may be hints we might want to add in future and hence a Hint structure identifies a type and value StartPoint hint can support 4 modes. These modes are only applicable when the HintType is StartPoint Interval hint is totally separate hintType that defines ranges of groups/objects
Having a registry is the idea and I was thinking once we get to agreement of Hint strucutres , then we can follow with defining error codes, registries for types and cleaning up track request params
Payload(b) is defined to have len and value . Here is the definition as defined in the draft today x (b): : Indicates that x consists of a variable length integer, followed by that many bytes of binary data.
So pushed few more commits cleaning up somethings and considering suggestion from Alan on TrackOffset and removing text on error. Thanks everyone for the review feedback Would love to hear feedback on the whole text.
Given the length of 245. Could you please summarize the use-cases to help focus the discussion. In the order of small to big, my suggestion of the order would be "3, 4, 5, 2, 1 "
There are 4 examples right at the end of 245, I copied them in this review comment: URL We can discuss tomorrow.
Alan proposed we use 6 parameters. Start Object Start Group Start relative / absolute flag End Object End Group End relative / absolute flag Preference for not two ways to say same thing. Use 0 to indicate current. Offset encoding. You can subscribe to the track twice with two different hints if that is what you want. Seems good consensus on call for this. Further conversation got to the point that this fails to be able to do both the first object of current group and current object in current group. So we need to do that too. Update on this luke porposed make rel/abs flag for both object and group which solves this
My proposal is 4 parameters, combining the flag with the value: Start Group + Relative/Absolute Flag Start Object + Relative/Absolute Flag End Group + Relative/Absolute Flag End Object + Relative/Absolute Flag I'm using the lower bits of the VarInt to save some bytes and make parsing easier. Otherwise you have to deal with missing the corresponding sequence/mode parameter. I think it's even easier if these 4 VarInts are just encoded as part of SUBSCRIBE instead of being parameters. There would be a None flag if you don't want an end group and/or object.
+1 on making it part of Subscribe.
Regardless of the approach, I think these are the use-cases we MUST support: I think these are use-cases worth considering: And not mentioned in this PR, but I do think we need UNSUBSCRIBE hints.
New proposal: Range requests would be accomplished with a SUBSCRIBE followed by an immediate UNSUBSCRIBE. This is a race though, and a reason to not take this approach. Keep-alive would be kinda funny if only the latest UNSUBSCRIBE is active.
In addition to what is summarized above, we had a discussion about the way to express relative ranges. In the case of the start hint, the well understood scenarios that I heard are: start at a specific point (specific group, specific object) start at the current point (current group, current object) start at the beginning of the current group (e.g., because the receiver needs all the frames from the beginning of the current group to synchronize its decoder) start at the beginning of the next group (e.g., because the receiver does not want to process old frames, and would rather start clean on a group start) We can make it more complicated, and we have proposals such as: start N (groups, objects) before the current one (in the past) start N (groups, objects) after the current one (in the future). I think that these relative offsets bring a lot of complexity, and a lot of corner cases. They are somewhat undefined, because the subscriber does not know what the current group or object is. If it knew, it could just ask for a specific start point. The combination of groups and objects is a bit weird, because the subscriber does not know how many objects there are in the past groups. The syntax allows for silly requests, like "start 1,000,000 groups in the past", or requests that are latent DOS attack like "start 1,000,000 groups in the future", which means "keep state in memory forever". There is a potential simplification there...
I want to keep the scope simple and focussed. When i presented on Subscription hints, there were few real use-cases that had a need for hints User joining a mid call and want to start from the beginning of the group to catchup or wait until next group User rejoins mid call and want wherever the current state of group/object is User disconnected and joining at a known point and want to catch up from there ( say to 2x catchup render) User wants to buffer a few seconds from past to keep the player go smoothly User asking for range of frames and be done with it as catch up Can we define a system that works for all possible combinations ever - Oh YEAH .. But do we really need it - I DOUBT IT I am waiting on Alan's proposal for the promised simplification. NAME no pressure :-)
I was pretty busy today but have been written up what we discussed during the call - using 6 track request parameters offering the full range of options that are available in this PR as is. By using defaults when a param is omitted, there was only one invalid combination, which I like. The logic to determine what group and object to actually start serving is about 20 lines of pseudo code, which I added as an appendix. I'm not sure what to make of the discussion on this PR. I'll write a second PR that is simpler and just includes the 4 basic modes, and maybe something for range. Stay tuned. Would it be better to submit both as separate PR's, and leave this proposal as is, or should I push what we discussed this morning here?
Its ok to push here .. how easy would it be merge both into a single commit ?
Commenting on Luke's table of use cases: Love this way of enumerating and sorting the use cases. Thanks. All the use case I think are critical are covered in the first table and nothing that I did not care about seemed like it was going to complicated implementations so my first reaction was YES. But looking at it, we were getting close to covering 100% of all the cases and since relays are likely to upgrade slower than clients, I just came down to we should just support all the possible combinations even if it is not clear there is s use case for it today - that will actually make implementations simpler. I think the proposed solutions are starting to look like they do all the cases anyways. By all the cases, I mean a subscribe can Start at absolute or relative for both group and object. Relative can have delta to current. Start at absolute or relative or fin for both group and object. Relative can have delta to current.
On the topic of do we need unsubscribe hints .... How I think of it is subscribe installs some state on the relay or publisher telling it what to do. It can say what the first and last object is that should be forwarded. However, I think as unsubscribe as the message that removes all that state. If we start making unsubscribe be "forward until you see object X" then remove all the state, we get into questions like what happens if object X-1 arrives after X. That said, I do agree with the use case that Luke is getting at of things like. Client was subscribed to something forever, but now I only want to be subscribe up to the end of the current group so that the client can do a clean switch over to some different subscription at the group boundary
I was thinking about use case for things like text media some of the stuff with the game moves draft that are not audio and video. It is pretty common to want to be able to get the last N things in a group first then later go get the earlier things in a group. The usual rest style pagination type thing where. Imagine you had a group where each object was a message in a chat, you might only want the most recent 10 messages but then when user scrolled back, to get the rest. The more I thought about this, I think we have a requirement for start group notes -- -- absolute Get last N object in a given group
Yeah, I'm thinking we should use RESUBSCRIBE instead of adding hints to UNSUBSCRIBE. UNSUBSCRIBE would always be an immediate and cleanup any state, while RESUBSCRIBE would be used to update the start/end bounds.
I took the liberty of adding a 3rd proposal to the PR but did not change anything else in the PR.
+1 - that does seem like it will be a very good way to deal with it
NAME I wrote this up on my phone before noticing your commit. It's basically your proposal 3, but not using optional parameters. It would be encoded as: I really like that Alan wrote some pseudo-code, although I agree that it shouldn't be in the final draft. The devil is in the details with this encoding, and I think this scheme is both the simplest and the most powerful. It's also the smallest on the wire if that's any consolation prize.
And I thought it would be a bug to specify startobject=none with my encoding, but it actually seems like a valid use-case. start group notes -- -- none Check if a track exists You would get a SUBSCRIBEOK if the track exists, but the subscription won't actually do anything. It kinda seems useful for specifying intent too. start group notes -- -- N Start warning the cache for group N
My proposal in one of the authors call was Subscribe Update message.
So NAME proposal in the thread above that is similar to mine just seems better than what I proposed. It covers all the cases, it is easy to describe and validate work and does not have a bunch of uses cases. I would put in a very weak argument for instead of taking the bottom two bits to encode the (absolute, relative, future, none ) , instead just put that into it's own VarInt. I would not argue very much for that because this is all going to get put in 5 lines of code I will never look at again and it's does not turn up on the wire in something that has to be small. Probably the worst part of is is someone looking at network dump might be confused but that just means the wireshark dissector needs to know about how to the >>2 works. I'd be totally fine with proposal as is. Some strong points I see for this: 1) we could merge it by Monday 2) It covers all the uses case we can imagine now plus has a high chance of covering the future cases we can not imagine now 3) It is simple to understand, think about, and test
A 0-16 range would work for the majority of relative/future hints, so it would definitely save that byte. But I'm fine keeping it verbose for now; the important part is the reusable encoding. would be useful for checking track existence without receiving OBJECTs. But let's rule it out for now because it would mean a subscription that will never match objects, which would need some extra text. would be same as but that's not obvious. We can also rule it out. is required to specify the FIN of a group. ex. end at the last object of group 69. is invalid though. I think it's a hint right now, since the publisher can choose to drop objects/groups on a whim. Even if you ask to start at group 3, you might never get it for congestion control reasons. But the bigger issue is that a subscriber doesn't know the upstream's expiration time. A publisher may only cache the last 3 groups, so a subscriber can't just ask for the the last 8 groups and expect to receive them all. Throwing an error in that situation seems futile. But I think the publisher should signal the (absolute) start/end range in SUBSCRIBEOK, which may be a subset of the requested range. Then it would be a start/end range request, resulting in a start/end range response.
I like the direction that Alan has take Lukes stuff. To bikeshed the Hint. I think "Location" would be a better name than "Hint". I think this is not really a hint but more a filter that relay will not send me objects on this subscription before the Start location or after the end Location. So if you want my bikeshedy 0.000002 cents, I would define Location { GroupMode (i), GroupValue (i), ObjectMode (i), ObjectValue (i) } ObjectModes = None, Absolute, Past, Future GroupModes = None, Absolute, Past, Future The the subscribe would have a Start and End Location. Location become a clean way to specify a specific a specific object in a track and may turn out to be useful in other parts of the spec.
I'd like to take a moment to reflect that the notation we are using for is both shitty and bad. Take Hint { Mode (i), [Value (i)] } Modes = None, Relative, Next, Absolute By shitty, I mean super non intuitive. What does (i) mean? Is [X] optional or an array? But no, that would be ... Can I have more than one X? If I had Mode(2) followed by [Value(i)], would it be bit packed or would the Mode be padded out to a byte? And what type is Mode. Or if Mode is a type what is then encoding of all that crap. Most schema languages are fairly intuitive to read and guess what they might mean. This is not. Then there is the bad. Say I want to define a type called Mode then have two fields in a record called ObjectMode and GroupMode both of type Mode. What do I do? Location { GroupMode (i), ObjectMode (i) } ObjectModes = None, Absolute, Past, Future GroupModes = None, Absolute, Past, Future This can not possibly be the right way to do it. I mean now that Mode is not reusable and repeats itself. If only we had a well defined language for expressing this. Oh wait we do URL
I went ahead and enumerated the permutations group notes -- absolute range request absolute tail of group absolute absolute full group relative head of live group relative tail of live group relative next audio sample(s) relative full group future head of future group future future future future group(s) none tail of track none The ones labeled with audio are only when you have partially independent samples. For example, each audio GROUP is 2s long and each audio OBJECT is 21ms. This would get you the last 105ms of audio (target initial buffer) most of the time. However, you could get unlucky and a new group could have started with less than that available, but it doesn't really matter if the target buffer is small because the player will just wait. Personally, I want to remove this use-case. If objects can be independent, then they should be separate groups IMO. So each audio frame would be a group of size 21ms. You could then combine the group/object mode since object boundaries can either be absolute or none.
I pushed a new commit with PROPOSAL 4, which we have been discussing, pulling the "best of" text and examples from the other proposals. I deleted the PROPOSAL 1 sample code and didn't add code for 4. There's one DISCUSS, whether we should allow Start/EndObject to be None and have defaults, or require them to be specified.
Currently, we have GROUP SEQUENCE and OBJECT SEQUENCE parameters that allow subscribing to a track at a specific offset. As far as I can tell, we do not actually describe what happens when those are not present. I imagine what most subscribers would want is to receive most recent N groups when they subscribe (for some subscriber-specified N that is determined by how large they want their buffer to be), and we should add a parameter for the said N.
This is a special case of Subscription hint issue IIUC () Wondering if we should discuss together
I hit this in moq-chat when traversing a non-caching relay - a client's subscription to a catalog track came after the most recent group/object was published, so the client didn't get a catalog. Do all relays need to implement caching of some kind in order to satisfy late-comer requests? Or something else?
I think the default behavior should be to cache/serve the latest group. It's basically required for any sort of catalog. But I do think we should support cache-less senders. They can basically only support from and will ignore requests to start subscriptions further back.
Subscribers needs a way to hint the publishers on when do they have the want the objects to delivered to them within the requested track. This is needed in case due to the fact that different media types can be handled differently by the application ( audio can be delivered to the application immediately, video needs some form of key-frame to have the best experience or something more specific that the application wants ) In that effect , i want to propose an experimental extension to the Track Request Parameters called "SubscriptionHint" with the following variants. latest (0): The "latest" subscription hint identifies delivery of the objects for the requested track to begin from the most recent group and object sequence. catchup (1): The "catchup" subscription hint identifies delivery of the objects for the requested track from the beginning of the group provided in the GroupSequence track request param. waitup(2): The "waitup" subscription hint identifies delivery of the objects for the requested track from the beginning of the new group.
One of the reason this is key, is often a client is doing a subscribe to a track with a different bandwidth because there is not enough bandwidth for the current track. ( the case where a client is receiving a high bandwidth track, and wants to move a low bandwidth track of the same stuff ). In this case, there is already congestion happening and the client wants to move to the new track quickly without making the congestion worse. Hints along the lines of this really help in this critical case.
Oof yeah this is a difficult problem that we need to solve. A new player can start X seconds behind the live playhead, where X is the target size of the jitter buffer. Real-time will set this to 0. An existing player can switch between tracks at timestamp Y. (NAME example) Neither the player nor the relay knows the mapping from group timestamp. Tracks might not have aligned groups; different offsets, frequencies, boundaries, etc.
My issue with this proposal is that (0) latest and (2) waitup are based on the relay's cache and are extremely racey. The assumption is that the relay always has the latest group/object in cache. The reality is that it often won't, nor does it even know what the "latest" group is supposed to be. For example: A player is subscribed to 720p and encounters congestion after . It wants to subscribe to 360p and it has three choices: If the latest object in cache is , then we receive some undecodable objects until starts. It's wasteful and exacerbate congestions, but hopefully for not too long. If the latest object in cache is , then is undecodable and and playback stalls until is generated in a few seconds (oof). If there's no cache, we could forward the SUBSCRIBE upstream with the and run into either of the two above scenarios. Note: the upstream may have it's own cache; it won't necessarily be the origin. If there's a partial cache, the "latest" object might actually be in the past. For example, another subscriber issued a SUBSCRIBE for 100ms ago, so our latest in cache might be . We incorrectly start delivering at and exacerbate congestion. (oof) Note that there's no way for a relay to know when it's transitioned from a partial cache (not caught up to live playhead) to a full cache (caught up to live playhead). Issuing two separate subscriptions, one for each mode, would be the only option, but that would temporarily double bandwidth usage (bad for first hop in particular). This works the best because it doesn't depend on what is in the relay's cache. It knows to always start at . However this requires groups to be aligned across different tracks (ex. HLS). If 720p does not correspond to 360p , then this isn't an option. It's also not an option for startup since the player doesn't know the latest group number. The player needs to somehow know which timestamps correspond to which group for each track. If the latest object in cache is , then we correctly wait until starts. If the latest object in cache is , then playback stalls until is generated in a few seconds (oof). If there's no cache, we could forward the SUBSCRIBE to the upstream and run into either of the two above scenarios. Note: the upstream may have it's own cache; it won't necessarily be the origin. If there's a partial cache, the "latest" object might actually be in the past. For example, another subscriber issued a SUBSCRIBE for 100ms ago, so our latest in cache might be . We incorrectly start delivering at and exacerbate congestion. (oof) Basically the same problems as (0), but on the other side of the coin flip.
That was a wall of text; hopefully I explained it well enough. Like I said initially, the fundamental issue is that neither the player nor the relay know the mapping from group timestamp. What if we resolved that? contains a timestamp field contains a relative/absolute timestamp. contains an optional timestamp for graceful termination. This suffers from some rounding/alignment issues but it's fine. Relative timestamp requests are also problematic with partial caches. track contains group/timestamp pairs. contains start group and optional end group. contains optional end group for graceful termination. This is absolutely the best option for relays. It knows what groups are missing in the cache and can fill them on request. It also means the relay can have multiple SUBSCRIBE requests upstream for the same track without fear of duplication. The problem is that the media protocol needs to produce a live group listing kinda like HLS. This incurs an extra RTT on startup, although it's super useful for seeking backwards and DVR.
I think there are scalability and flexibility benefits to keeping relays agnostic to time and in fact to any internal characteristics of the objects they are serving. Keeping them decoupled allows the content to change (and the complete timing mechanism if so desired), without having to change the relay networks. The introduction of relay’s requiring timestamp knowledge stems from the original assertion that I think this need not be true. We can teach players the relationship between group and timestamp without too much difficulty. We do this all the time with HLS and DASH. In both cases the HTTP servers delivering these formats know nothing about the relationship between segments and time, yet we are able to build players which seek to precise points in a stream. We can do something similar with our streaming formats and the ideas already presented in this thread by Suhas and Luke. One difference is that I would make would be to allow the group offset described by Suhas to be either an absolute or relative one. As a proposal, what if the WARP catalog listed two things: The GROUP duration in milliseconds. The name of a track which described GROUP number/timestamp pairs (exactly the track suggested by Luke above). We assume that this timeline track allows delta updates, perhaps describing the complete DVR window every minute at a group boundary, with incremental updates in between. Use cases: Start as close to the live edge as possible a. Since the last group represents the last independently accessible point in the stream, the player would issue a plain SUBSCRIBE b. The relay would serve content starting with the last group it has cached (its default behavior). c. If the relay does not have this content cached, and there is no pending subscription, then it would issue its own SUBSCRIBE upstream. Start with a 5s buffer a. The player would calculate how many group’s it would need to guarantee a 5s buffer. Assuming the advertised group duration was 2000, then it would need 3 groups. Using zeroth based relative group addressing, it would issue SUBSCRIBE (-2). b. The relay would serve from the 3rd last group that it had cached. c. If the relay does not have the content cached, it would send its own SUSBCRIBE(-2) upstream. If it has an open subscription, but has only received 2 groups, then it should wait for the 3rd before responding. Seek to a wall clock time of T seconds a. The player would subscribe to the /timelines track, using SUBSCRIBE so that it receives the last full DVR description. b. Based on the data in the timeline track, the player calculates that it needs a group number of 5321. It issues a SUBSCRIBE(5321). c. The relay recognizes that this is an absolute reference to a GROUP (as all relative ones are negative) and returns content starting at GROUP . Player encounters congestion at group 53 on 720p and wants to switch to 360p track a. WARP places the packaging restriction that group numbers across tracks must carry the same media time offset (it actually has this requirement already) b. The player knows it was playing group 53 when it encountered slowness. Its next chance for a switch will be at a group boundary. Since it knows the group duration from the catalog, it can decide whether it is better off waiting for the next group, or reloading from the same group. In this case it is near the end of the group, so it decides to fetch the next group. c. It issues a SUBSCRIBE (54) for the 360p track and a UNSUBSCRIBE(53) for the 720p. Cheers Will
Absolutely NAME The main difficulty with relative requests is deduplication: a relay has trouble merging subscriptions that use absolute and relative groups when the cache is empty. ex. SUBSCRIBE 480p -3 (50ms delay) SUBSCRIBE 480p 351 The relay doesn't know if the existing upstream subscription (-3) will fulfill both. In fact the relay doesn't know if 351 is even remotely close; it could be a group from minutes/hours ago. We don't want to download the same objects twice, but maybe that's okay when the cache is empty? I think we either: only use only absolute groups or add a way to OR an existing SUBSCRIBE
Thinking about it some more. If the player only knows the duration of each group (fixed), then it could use relative timestamps, but it's still a massive headache for the relay. Suppose two SUBSCRIBES from separate viewers: A relay with an empty cache issues the upstream. It starts getting some OBJECTs: . And then it gets a from a viewer. It would be an error to serve those three objects from cache. Why? Who knows if they were the latest three or from an hour ago. The relay could hypothetically augment the existing upstream subscription: . However, it still doesn't know which groups correspond to -3. No additional objects would be sent if 6330 actually was the live playhead, and the relay wouldn't know where to start the -3 subscribe. IMO the relay needs a way of converting relative groups to absolute groups for this to work. Maybe SUBSCRIBE_OK contains that translation? ex. . It's still complicated though.
You're indeed right. This the fatal weakness of the relative request approach. If the relay has the latest group cached and an active subscription, it works fine, but the player can never be sure of that and the relay can never disambiguate fresh from stale content. If the relay can do that translation, then why couldn't the player do the same and just request an absolute group to begin with? What are some solutions to this problem? Players always make absolute requests. Here are two ways to do this: a. The catalog instructs the player in how to calculate the group number for any given time, for example, via a start time and then modulus the group duration. To avoid players having to have an accurate clock, the SETUP response from the server could echo back the current time. This is essentially a DASH approach to timing. b. The player subscribes to the /timeline track which describes the wallclock time and media time associated with the start of each group. This is similar to an HLS approach and has the downside of requiring 1xRTT before starting to discover the latest group. It has an advantage that it can describe groups of varying duration and Each group carries a timestamp in its header indicating the UTC time at which it was produced. This is not violating any decoupling of relays and payload because the time of production is core characteristic of any payload and is independent of the internal media time. With these timestamps in place, the relay can judge freshness. A player can issue SUBSCRIBE (-5) which means gives me this track starting with any group whose timestamp T > (now -5). We could also consider SUBSCRIBE (from: 1695571105212), which would require the player to have an accurate clock (although SETUP could be used to address that). Note that this scheme also works for DVR seeking if the player knows how to translate group number to UTC time. Same as [2] except that we add both a media timestamp and a UTC timestamp to each group. This gives the flexibility to ask relays to begin subscriptions at both media time offset and absolute and relative UTC offsets. The downsides are that it binds moq-transport to internal properties of payloads. One way round this is in core moq-transport we can specify two "index fields" in the header of each group. These fields must be numeric and monotonically increasing. Relays don't know that these fields represent, however they can service subscription requests that reference them. The streaming format would then bind index A to UTC time and index B to media time. Other formats may bind different attributes, but all could still work with generic relays. After writing these out , I'm in favor of [3] with generic indexes.
Yeah exactly. UTC timestamps get annoying because of clock synchronization. It's also annoying because the player has to know the latency to the origin; requesting media that was generated 200ms ago results in a very different experience if the origin is 10 miles away versus 10k miles away. I like the idea of delegating the group timestamp mapping to the application. The catalog could contain a formula (like DASH) or it could maintain a /timeline track. That being said, we still run into the same issue; how are you supposed to request the latest group in the /timeline track? Do you always request ? And is the latest, or do you need to wait until you receive the full /timeline? I think there's a missing piece. If I were to implement this in HTTP, I would make a GET request with an ETAG to get the latest group/object sequence. There's no equivalent in MoQ which is problematic. I think we still need relative group requests, but the relays need an authoritative source somehow; it can't decide what is "latest" from cache.
I think that's not what it meant. By "latest", relays starts delivering whatever it has in the cache as the most recent group/object and it doesn't need to remember anything about it other than they happen to be in cache. "waitup" is saying given everything once you get objects from current group +1.
+1 on keeping the relays not aware of media characteristics. Since its complex and may not apply the same way to all the types of media getting transported (live vs dvr vs chat vs catalog vs something new). +1 to keep players aware of mapping from media time to group number. I would be interested in knowing where this will not work though. I agree with Will that the warp catalog can specify the needed relationship here Also the susbcribe message today has "Group/Object Id" in its track request parameters. I would proposes merging Will's usecases (2) and (3) into given a Time T, find the group number you want to start fetching the media Issue a subscribe with that group number Relay can decide if the requsted group number is behind or ahead of the current group and if it is in cache serve them accordingly Basically i think we don't need offset. But would love to hear from NAME where we can't do away with absolute groups
We need to be clear on the term latest. In my proposal of Subscription Hints "Latest" is in terms of latest in the cache. So if the cache is current running group-45, object-32, then that's the latest. And waitup is , wait until you see group number change before giving me objects. I think with the hints listed, groupid & objectid being in the track request parameters and player providing the mapping function from timeline to group, we can address most of the use-cases being raised here. I would interested to learn what use-cases cannot be addressed with the proposed scheme
Can you give an example using the "latest" mode? Would you use it for playback start or switching between tracks or both? As I understand, groups are independent but objects are not. would be a P-frame or catalog delta. I don't understand why you would ever want to start your subscription at an undecodable object. Starting at or would work, but never .
For the video delivery case, I would use "waitup" if i have no idea what the current group is. Since it will start delivery the objects from the beginning of the next group. I would use latest for audio use-case where the is fine for me to either render or conceal. Again we can enumerate the use-cases we want to address and I am happy to share my thinking around how the hints help. As part of exercise , we may come out deciding if we need more granular things than what is being proposed here. NAME could you please share one liner for the use-cases you have been considering from a subscribe perspective ?
The key point we're forgetting in the freshness quandary (or perhaps it is implicit) is that the relay knows whether it has an open subscription for a given track. This is a key bit of information. If it does have an open subscription, then the objects it has in cache are by definition the latest, and hence any 'relative' or 'latest' queries can be satisfied. If it does not have an open subscription, then it cannot make any determination about the freshness of any objects, even if it has some in cache and it must renew its subscription before responding. To illustrate this, consider a relay which has groups 637,638,639 in cache. Five minutes later another client joins issues a SUBSCRIBE (latest). The response of the relay should be different in the two cases below: It has an open subscription for that track - in this case it knows that those are still the latest objects available for that track and so it would return 639 It does not have an open subscription for that track (because the prior client that subscribed went away) - in this case the relay must make a SUBSCRIBE(latest) request upstream and return the first group it receives (perhaps 763) to the client. The advantage of relative requests is that the player does not need any understanding of the relationship between group number and time. This makes for simple players, as long as they don't need to seek behind live with any accuracy. The relative datum, in this case the "live edge", is constantly changing, and for players that need to seek back accurately this can be a problem. Depending on when you make your request and how long it is queued by intermediaries, you may get two different answers. Absolute groups make precise group selection for more deterministic and are robust. For live-only players (such as the web meeting use case), we probably could get by with relative addressing alone. However since our charter includes intermediate and higher levels of latency, we should design moq-transport such that it is flexible enough to handle both relative and absolute group addressing.
if there are no upstream subscriptions active and hence no published data is being received , then there are couple of things to note What's in the cache is the current source of truth and it needs to be served yes Relay may forward the subscription upstream and feed the data that arrives towards the subscriber ( and also add it to the cache) This will keep "latest" property preserved. Players need to handle gaps that might be caused for several reasons including relay cache purges, publisher going away/reconnecting, subscribers loosing state, relay reboots. I see these as player level details and not specific to moqt. Isn't it ?
It's not a source of truth at all. It may be 5 minutes old or 50 minutes old. Without the active subscription, there is no guarantee about its status at all other than it was previously served to some client and hasn't yet fallen out of cache. Why would a client wanting "latest" want that and if it was sent, how would the client know that it wasn't the latest. It would start rendering and then jump ahead when the true latest content arrived . If you need a subscription-hint for give-me-anything-in-your-cache-not-matter-how-old-it-is, we can create that, but thats different from "latest" and I'm not clear on its utility.
In particular, I would like to see MoQ also support some improvements for DVR or even VoD playback, for example: faster frame-accurate seeking. We don't necessarily need the protocol to provide a complete implementation, but having the ability to absolutely address groups would be especially useful in that context. One concern I have about adding timestamps into the mix is in how they would apply to content being played back at arbitrary points in time relative to the original recording time.
Here's my current thought. SUBSCRIBE can contain a relative or absolute start group. A bit flag is used to specify if it's an absolute or relative request. N means serve group N and onwards +1 means serve the next latest group (waitup) -0 means serve the latest group (catchup) -N means serve N groups prior to latest SUBSCRIBEOK contains the latest and . Useful for players: it knows it should start rendering after decoding object=K Useful for relays: enough information to serve relative requests from cache. Relays can compute "latest" if it has an active subscription: if SUBSCRIBE active: if SUBSCRIBE inactive: You can serve relative subscriptions from cache only once you have an active SUBSCRIBE and the corresponding SUBSCRIBEOK. If you don't have an active subscription, then you need to SUBSCRIBE again. The SUBSCRIBEOK will tell the relay how stale the cache is and what can still be served.
Since URL can only be equal to but never greater than URL, this simplifies to Another another topic, we should also allow the use case of subscribing to a stream before any groups have been produced. In this case we need some convention to convey that no groups are yet available, for example group=null and object=null
Nah, is updated on receipt of a new object. It's used for any new relative subscriptions. Here's a full example from the relay <- relay POV. The edge receives a SUBSCRIBE -3 from a viewer or downstream and has an empty cache: The subscription hasn't caught up to the live edge yet, but now the relay has a snapshot of how to map relative requests to absolute requests. If the relay receives a relative SUBSCRIBE during this state, it will know what can be served from cache. For example, a new SUBSCRIBE asking for 0 will not receive 6743 thanks to this hint. It's the only object in cache at the moment, so it would have been served with or but not with this approach. The SUBSCRIBEOK says that 6745 is the latest so we just have to wait for the data to finish transfering. This snapshot of latest will go stale, but the active subscribe will catch up soon. The snapshot exists mainly to support the first ~100ms where the subscribe is backfilling. In the meantime, the relay keeps receiving groups, possibly out of order: Upon receiving 6746, the relay updates the value since it's now larger than the snapshot from SUBSCRIBEOK. A new downstream SUBSCRIBE asking for -1 now will get 6745 and 6746 from cache. Note that there's no new upstream SUBSCRIBE; we can continue to serve relative requests indefinitely from cache unless they are too far in the past. Just to complete the example, let's suppose the relay UNSUBSCRIBEs right now for whatever reason. 6742-6746 are still in cache but we've wiped our "latest" value since the subscription is no longer active. The relay could get either a new absolute or relative downstream subscriber: : the relay serves 6745-6746 from cache while SUBSCRIBE 6747 is sent upstream. : the relay doesn't know if it can serve from cache since the active subscription has ended. The is sent upstream and the relay behavior depends on the resulting : : both 6745 and 6746 are served from cache : only 6746 can be served from cache : the cache is stale and can't be used In the first two cases, the relay would receive a duplicate copy of the objects it already has cached. That's probably fine, but a way to avoid this would be to subscribe based on both absolute and relative groups:
Please do not use these tricks of encoding meaning as the sign of an integer. It looks like a premature optimization, and in fact it is not: there is no syntax defined to encode negative integer, you will end up with a complement to 2 64 bit integer encoded on 8 bytes. It will be much cleaner and also more efficient to encode the hint as a tuple: meaning: an enum value describing what the subscriber wants, such as start at the current group, the next group, the first group, a specific group, or a specific location. additional parameters if needed, such as actual group ID if not "current", "next" or "first", and actual object ID if needs to be specified.
Consensus at the interim is that we definitely want this and we both want to relative (n groups back) and range requests (ie: group n to group m), which could be open-ended. Joining from the currently ongoing group (startpoint=-1 in slides), and the next group (startpoint=1) had strong interest. start_point=0 starts from the current object within a group and there is contention about whether it's necessary given groups are intended as start points, but there was enough interest in the room that it was felt the correct way forward was to support it and then we can revisit the issue in the future if needed. Additionally, there is interest in object granularity, particularly for the range requests. A solution should and .
+1 to christian and happy to help with the PR . thanks
Here's what I was thinking: These fields could be optional via a parameter, message type flag, or a . My preference would be the mode because it's easier to decode and encode. It would take 4 bytes to specify all of the start/end boundaries versus up to 12 bytes if they're all parameters. Here's how the modes from would map: I think this is simpler and more powerful. A publisher just needs to write a function to convert a relative boundary into an absolute one based on the latest value in the cache. This might be an asynchronous process for a relay if the cache is empty or new. Here's some things you can't do with the proposed encoding in : Start 4 groups in the past and end 2 in the future: Start at group 69 and end at latest: Start at 69 and end 4 groups in the past (might be a noop): Keep refreshing a relative subscription while it's needed:
Some use-case like catchup , go back to a given object for layer refresh or server the game state from a given object , and similar scenarios , would need a name to ask for the object. It would be good for the protocol to be able to subscribe to not just track and group, but also object.
As an individual: +1. An application should be able to request a single object. Maybe even a byte range of an object but I would punt that until an actual use case materializes.
Can you elaborate on the use-case? Resumption after a connection is severed makes sense. We need to solve that for both the contribution and distribution use-case. NAME has some strong opinions here. I don't quite understand the second use-case. Why would a consumer need to refetch a single object, but none of the objects it could depend on? It only makes sense when there's a hole in the group, but why did that happen and how does the subscriber know about it?
And to be clear, I also think we should have a GET equivalent, but I want to make sure I understand the ask.
The encoder might have skipped some frames due to input loss or another issue, but maybe they are available from a redundant encoder and you would like to "fill the gaps"? The subscriber only sees the gap, it does not need to know the dependency structure.
Two uses cases I am thinking, both on distribution side: Use case A: client join a session that is in progress and wants to join Track A. The client wants to start playing before the next the GOP starts so it subscribes to Track A with the most recent group and wants to get all the objects in the group. Use case B: client is watching Track A and has received all the objects up to group 123 object 45 and looses wifi and switched to 5G. At this point it forms a new subscription to Track A but it wants to start at group 123 object 46 because it already has the stuff before that. If it had to wait to received all the stuff in group 123 over again, it would add delay as it switched networks. There are some more uses from scalable codecs but if we can solve the above two cases, we will probably solve the toher uses cases as well.
I think the solution to this is simply that the Subscribe can have some sort of flag that indicates if the clients wants just future objects, or if wants to catch up on all the objects in the current group, or wants all the objects after some particular objectID
There isn't a way to GET a single object - you can SUBSCRIBE to the track starting from that object, and perhaps issues an UNSUBSCRIBE after that. I'm going to leave this open for now.
Will suggested solving this as part of and have both a start group and end group.
Merge it :-)


Feel free to change the numbers to any ones you prefer.
There's a single SETUP message, but the contents change based on if the endpoint is a server or a client. . This is difficult to implement in type safe languages. I would implement and , sharing the same ID. However, it's then difficult to implement a generic method as it needs to know the expected role to distinguish between the two. I propose a separate and message akin to how TLS has a separate and . Alternatively, implement to reclassify SETUP and OBJECT as control and data stream headers respectively.
Hmm, so this isn't a problem in my code any longer since made SETUP a special message type. We should still figure out if both the client and server SETUP use the same ID, or separate IDs like TLS.
I still like the idea of, if these message are different, they should have different names as it will just be confusing if they do not.
Agreement these should be different message types, just like TLS.
Conveniently, message id=2 is currently not used. I propose ClientSetup = 1 and ServerSetup = 2.
Thinking about this a bit more, do we want to bake in the assumption that the client always speaks first into the protocol? For raw QUIC, typically, the server would have 1-RTT keys first.
I would love to see something like HTTP/3 SETTINGS where both endpoints can send the message in parallel. That requires version negotiation at a lower level in something like ALPN though. The SERVER SETUP would be sent first in native QUIC while the CLIENT SETUP could be sent first with WebTransport (in parallel with CONNECT).
LGTM


This were replaced in Subscribe Locations (hints)
These parameters have been replaced with explicit fields in the SUBSCRIBE message. All that's left (outside of SETUP) is AUTHORIZATIONINFO in SUBSCRIBE and ANNOUNCE. I would just as soon get rid of non-setup parameters entirely and have an "authinfolength" field in the messages which can be zero. At a minimum, we need to eliminate GROUP and OBJECT_SEQUENCE.
+1 I would go even further and make a dedicated AUTH message.
Whoops, those were deleted but must have been re-added in a merge conflict.


When we added (b) we didn't update this.
SUBSCRIBE_RST as well
and ANNOUNCE_ERROR


This PR is expected to be base PR to support multi-subscribe like flows that will follow up. It adds support for SubscribeID and also renames the trackId to to avoid the confusion and make the scope clear.
NAME NAME NAME wondering if the latest commits address the feedback. thanks
The normative bits look fine. PR needs some cleanup. Recommend splitting ID -> Alias to another PR, so this subscribe ID doesn't get caught up with bikeshedding the field name.LGTM but I would like Subscribe ID to be the first field in every message for consistency.Still looks fine after the split Having reviewed some code that does subscription matching with the current draft, I strongly endorse splitting those.


I also wonder if Connection URL should be renamed to Session URL?
Part of PR has text "MUST close the connection if there are duplicates" It seems like we should be closing the session, and we should specify an error code.
I think this is editorial, but unsure because of "specify an error code"
I would prefer: instead of:


I believe some of this is covered by the recently updated GOAWAY section and I'm not sure what else to add here.
I don't think these TODOs have been covered yet. That being said, I would rather we file issues for these so there can be a conversation, rather than litter the draft with TODOs.
I'm unsure what's unresolved, but if you can file new issues as appropriate and link them to this PR, that would be helpful.


I think the intent of the draft was clear but this makes it explicit not to open a uni stream before setup exchange is done.
It's unclear what messages are allowed to be sent or received during the SETUP handshake. Here's my intuition: Until receiving the SETUP, the client: MUST NOT send messages MUST buffer any received messages. Until receiving the SETUP, the server: MUST NOT send messages. MUST NOT receive messages. The client needs to buffer messages because the server is unaware of when the client receives the SETUP. There's no equivalent to HANDSHAKE_DONE, and adding something like that would add an RTT anyway.
A potential optimization exists if the message encoding if the same for all supported versions (invariants). For example, if the client only advertises support for versions 1,2,3 and those versions share the same SUBSCRIBE encoding and meaning. This seems unlikely, given a separate version was created in the first place, and seems very brittle. If this was the case, then the client could process any invariant messages immediately and buffer the others. However, the client still couldn't send messages unless the server buffered any received messages.
The most flexible optimization would be the one where you could encode the SUBSCRIBE_REQUEST for every version you offer as a field in SETUP parameters. TLS does not do exactly that, but there are some things in ClinetHello that are only applicable for certain versions.


Thanks NAME - all the proposed changes look like definite improvements. Thanks I need more coffee today.
In HTTP, there are a number of restrictions on what characters are allowed in a path, headers, etc. I don't see any text restricting track names and track namespaces, but the examples are all ASCII, implying the namespace and name are not intended to be completely opaque and are instead intended to be human readable. There's nothing in the spec that prohibits them being treated as opaque identifiers, but I'm unclear if that's the intent and I think it'd be good to explicitly say what we intend.
Most of the restrictions in the HTTP world are derivative of the restrictions in URIs: see RFC 3986, sections 2 and 3. I'd suggest we adopt those limitations and the percent encoding method from HTTP. That allows folks to use a well-known set of methods and aligns with the idea that we might have one or more URI schemes in the MoQ protocol space.
I believe we currently don't have a way to put a track name or a track namespace into a URL, so I don't think those restrictions apply.
I agree that they don't apply now, but if we adopt them it will simplify the set we have to deal with (since we will have the same set as other parts of the protocol). I suspect people will want the namespaces to be easy to generate from FQDNs (which are also the authority part of HTTP URIs), as an example.
IMO a track name should be human readable. Otherwise we should use numeric IDs instead, like most containers. I also think the track namespace should be human readable since it will likely be exposed to the user. Currently you need a URL and track namespace to configure a player/broadcaster. It will be quite common to combine both into a single URL (ex. like my relay) so it would be nice if track namespaces could be encoded into URLs.
Maybe we should ask the flip side: is there a use-case that won't work if track names are utf-8?
Moq transport must not define the interpretation. At the moqt transport layer track name and track namespace are set of octets and relay perform fast match where needed. How things from application domain gets mapped and gets setup with Relay is out of scope of MoQ Transport. Also how an application sets up namespaces with CDN/Relays is out of scope of moq.
We should clarify in the spec to say these are opaque and examples are intended for human readability purposes only.
I don't have any particular heartburn about where this gets defined, but I don't believe that the overall system will function well if the namespaces and track names are defined solely as "octets". We are talking about these as a tuple of namespace and name to create a globally unique identifier. If there isn't a common agreement on how those are composed and of what they are constructed, different systems doing different things will hinder interoperability and/or generate collisions. I would personally prefer the HTTP syntax, but UTF-8 would likely be okay if we specify how to handle things like byte-order marks, joiners, bidi, arabic letter marks and so on. That's a lot of work, but if someone wants to do it, okay. I don't think accepting a bunch of different encodings is helpful and I don't think using non-human readable identifiers is going to fly without a mapping to them (which only shoves the problem up a layer of indirection and adds in something new to fail). If not moqt, where do you want all this defined?
I disagree. We have to recognize that we are distributing the process of minting these identifiers very widely. Allowing people to re-use existing systems to create these is going to get us better scale and present fewer problems. You should expect folks to re-use DNS names to mint namespaces, to take an obvious example.
I don't think I am forbidding any of these. I am just saying it is not something moqt should enforce or we should support all forms of these not just one class of things
I am also strongly in favor of a utf-8 restriction. I'd like to inherit the usability achieved from a globally deployed and understood resource identifiers and to leverage the existing DNS and certificate system for authentication and routing. If we define names in terms of octets then we need to reinvent these conventions, or else have an unnecessarily bi-furcated system in which some addressing occurs via binary octets and others occurs via utf-8 resource locators.
The AUTH parameter is currently defined as an ASCII string. Whatever we use for track names, can the auth token share the same encoding?
Summary of IETF 118 meeting is there are no restrictions on at the moq transport layer but there may be a layers above this. This is to say the compare function for track namespace and track name are the equivalent of posix memcmp.
Looks fine to me. Agree with most of the nits suggested by Ian


It's only used a few times and now that Namespace and Name are an opaque sequence of bytes, it's even less compelling.
Take the line from your drafts that says There is no valid way to parse the right hand of this. We have things like "Full Track Name Length" and "Full Track Name". I do not think we should have ambiguous syntax for things that require precision. I think it would also be good to have things that can be directly mapped to common programming languages. This moves beyound simply editorial, or specification is underspecified and ambiguous until we fix this.
One possible solution would be use underscore or CamelCase
Oh, and the longer we leave this dumpster fire, the more painful it will be to fix.
love it


This PR brings in currently active proposals for supporting explicit indicators of mapping object model to quic transport.
Please keep discussion of the concepts on the issue
If there are any other proposed solutions to this, would be great to have them documented as additional proposal in this PR.
My recommendation is to pause a bit on this PR. We need more normative language, explanatory prose, and examples to help readers understand and evaluate the proposals without breaking github with a long chain of questions and comments. The outcome from Monday's meeting was that the chairs were going to designate a small group to put the proposal forward. I'll circle with Ian and Ted.
OK - will wait for that
This is all just so confused. The number of messages, modes, and permutations are exploding. Some ideas are in the right direction (ex. stream header) but the hodgepodge of modes/messages is not. I'd like to explicitly add proposal 4: a relay forwards data streams. when a relay receives an upstream stream, it creates a downstream stream (for each subscriber) when a relay reads an OBJECT from an upstream, it writes it to the corresponding downstream if it matches the SUBSCRIBE criteria. when the upstream is closed, the corresponding downstreams are closed with the same error code or FIN. when a relay receives an upstream DATAGRAM, it sends it downstream if it matches the SUBSCRIBE criteria. No wire changes. I would like a stream header eventually but it's more for compression and to prevent foot guns (ex. prioritization). This unlocks experimentation as the application can perform any stream mapping. The relay doesn't care and remains simple. The important part is the guarantee that the relay (middlebox) won't rearrange or reorder shit if the application decided to use a stream. edit: oops was logged into my work account.
The consensus in Prague was to move forward with a proposal that uses explicit signals for now, get more implementation experience and data, and revisit. Proposal 4 only uses implicit signals from the QUIC layer rather than explicit signals from the MoQ layer. I think you can achieve similar outcomes with proposal 3 by adding the appropriate STREAM_HEADER at the beginning of each stream?
+1 to NAME proposal. once we clean it up by removing the other proposals , we should be good for merge review. also +1 on dropping the priority for now
Oh and we should be some text: Otherwise we don't fix the core issue, which is that relays can arbitrarily change how OBJECTs are delivered. Stream mapping is useless if it's not enforced.
3 is a nice refactor of approach in 1. I like it, expect for two things: I don't think we need to support track-per-stream, and I don't remember if we ever actually got consensus for supporting it (we might need to discuss this separately). Priority/send order should be a stream-global, since that's the level at which those are determined in QUIC. The text is not entirely clear to me: is it still legal to have two open streams for the same group with STREAMHEADERGROUP (with understanding that they will get merged at the relay)? I think we could make approach 4 that does to 2 the same things that 3 does to 1. Canonical objects would have two fields: 'Object Placement Preference' that says that object is either (1) to be placed on a new stream, or (2) to be placed on the same stream as object N. 'Object Placement Close Bit' indicates whether there will be objects after this one on the stream or not (i.e. whether there is a FIN or not) There's one OBJECT_STREAM frame with Track Alias, Group Sequence, Priority / Send Order as headers. You can trivially figure out how to backfill those fields (you only need to remember the most recent object for each stream); the placement when retrieving from cache is similarly easy (you do need a map for "most recent object -> stream"). I think this fits in with the "implicit-explicit" approach that Luke has mentioned before. It also has a nice property that there are no cases in which what's on the wire can mismatch with what's in the canonical object (unless streams get randomly reset).
Track-per-stream would be the preferred delivery mode for VOD files. I'd very much like to ensure that we keep support for that use case open, even if most current implementations are focused on live.
I think it should be illegal as it breaks . The broadcaster uses stream mapping A but wants downstream relays to use stream mapping B, assuming there even is a relay involved. I don't think there's a use-case for this special treatment of the first-hop. The only use-case that I've heard is a broadcast client that is smart enough to detect group/object boundaries (ie. frames or key frames) to annotate each OBJECT with the correct stream mapping, but doesn't want to open multiple streams? That's a very poor reason, especially if the first hop may be congested, and the broadcast client should just use the indicated stream mapping. IMO a relay should not split or merge streams. It increases complexity and hurts performance. There's no need for your proposal 4 then.
Maybe. The assumption is a track-per-stream would allow you to write a simple VOD-only client. It wouldn't need to reorder or reassemble objects because they're all delivered in order on a single stream. That's optimistic, as it will still need to support potentially unordered/unreliable objects (like a live client) due to seeking and gaps in the recording. But I think we need to explore VOD and DVR more. I definitely don't want to require the server to modify the stream mapping at rest in order to serve VOD traffic, and it's not clear how that would even work for DVR. At the very least we should first support VOD with the original stream mapping. Much like how HLS can use the same segments for live, DVR, and VOD (segment size == stream mapping). And I think we should focus on DVR first.
This assumes LIVE-TO-VOD workflow. When I speak of VOD, its for pure VOD, which has no gaps, and was never live. The track-per-stream would would mean the client would never have to deal with out-of-order or lost objects, as the single stream would deliver them reliably and in order. Like you, I really like the appeal of the simple client interface for this. All I need is more space in my cache and I have a new market segment to address.
I'm confused. Proposal 4 does not allow merging or splitting streams. That is true, but VOD requires different delivery mechanism from what live does (e.g. live cares about catching up to the live state, but for VOD, you usually just want to show everything in order).
Recent discussions around the points highlighted in URL has made it clear that the core transport protocol needs to provide advices on how to moqt object model maps to the QUIC. Given that different use-cases have slightly varying requirements, there is also need for applications to specify their preference on how to use QUIC streams to transport objects and groups within a track from the publisher to the subscriber. Here is the proposal for enumeration as part of message as an experimental addition to the protocol for publishers to express their interest . Following delivery modes are proposed StreamPerObject(0): In this mode, the publisher intends to use one QUIC Stream per MOQT Object. Stream Per Group(1): In this mode, the publisher intends to use one QUIC Stream per MOQT Group. All the objects from a given Group share the same QUIC stream. Stream Per Priority(2): In this mode, the publisher intends to use one QUIC Stream per priority as determined by the MOQT object priority value. Stream Per Track(3): In this mode, the publisher intends to use one QUIC Stream for a given track. The core idea behind such experimental extension is to help under the application, deployment and interoperability requirements before the final path is chosen for the core transport
In moq-transport, an intermediary knows how to extract objects from incoming Streams (there may be 1..N Objects in a Stream), read their metadata (such as Group and priority assignments), cache them and potentially drop them depending on prioritzation settings. They also need to know how to forward them, which I believe is the intent of this issue. Perhaps we should reframe these as ? They instruct intermediaries beyond the current hop in how to propagate the objects . The various forwarding types can be enumerated in the spec and should be a new field in the header of each Object. If the intermediary is not making a new Stream for every Object it forwards, then it needs to bind a new Object to some existing Stream. I term this the "binding ID". I have relisted the groups, adding in the binding ID that an intermediary would use. Send each object in a new stream (default?) Binding ID: None needed. Send all objects with the same Group in the same stream Binding ID: namespace + Group number Send all objects with the same Priority in the same stream Binding ID: namespace + priority value Send all Objects in the same track in the same stream Binding ID: namespace + track name
I want something simpler. If a relay receives objects A, B on QUIC stream 1 and objects C, D on QUIC stream 2, then the relay continues that relationship. The relay MUST NOT reposition objects within a QUIC stream. Object per stream: A:1 B:2 C:3 D:4 Group per stream: AB:1 CD:2 Layer per stream: AC:1 BD:2 Track per stream ABCD:1 Bonus points if objects on the same stream share a header. ex. prioritization is applied at a stream granularity, not object granularity.
How would caching work under this scheme? Caching introduces a temporal discontinuity to the transmission. Should the relay continue to cache by Object, or should it instead cache by Stream?
You would cache by stream and byte offset, like HTTP chunked-transfer. If you evict A from the cache, you would also evict B from the cache. If the application chooses to put A+B on the same QUIC stream in that order, then it expects A+B to be delivered in that order. It the application wanted those objects to arrive in any order, then it would have put A and B on separate streams. Should we support a situation where a client transmits A+B on the same stream, but expects the relay to split them into separate A and B streams to downstream clients? I don't think so, because of congestion on the first mile. Once you introduce head-of-line blocking, you can't remove it. To oversimplify, it's like converting TCP -> UDP. If there's congestion on the TCP hop, then UDP packets will arrive in TCP order despite being split. This still has benefits for last-mile congestion versus TCP -> TCP, but to handle first-mile congestion what you really want is UDP -> UDP. Vice versa, should a client transmit A and B on separate streams, but the relay combines them into A+B for downstream clients? Absolutely not, as you actually suffer from both first-mile congestion and last-mile congestion. The analogy is UDP -> TCP. Any unreliable or out-of-order delivery at the first-mile needs to be fixed before it can be converted to reliable and ordered delivery. This is then exasperated by prioritization, as the relay wants object A to unblock B, but the sender may decide that C is higher priority. These same principals apply for RTMP -> WebRTC and WebRTC -> HLS. I spent years working on the former at Twitch and we gave up because the user experience was terrible. We offered WebRTC -> HLS but ironically it introduces more latency and is a worse user experience than RTMP -> HLS. I strongly believe that relays need to provide: X -> X If an application introduces head-of-line blocking by putting content on the same stream, then the relay must maintain that head-of-line blocking. The relay doesn't try to introduce more, nor does it try to introduce less. This is also amazingly simple for relay; they just proxy QUIC streams. The application still needs stream prioritization for inter-stream relationships, but now it can use stream ordering for intra-stream dependencies. For example, the base catalog and deltas would be on the same QUIC stream. There's no need for parent sequence number as the relay would deliver the base and deltas in order. There's no reason for the relay to cache them separately, as evicting the base but not the deltas from the cache would be a mistake.
NAME mentioned something at the start of the call that I wanted to address. It sounded like the client was transmitting a single QUIC stream, and he wanted parts of that stream to be cached independently. If this is a single GoP, then it can't be cached independently. If a new subscriber joins then it needs the entire GoP. You have to make multiple GoPs if you want independent join/cache points. If there are multiple GoPs sent over the same QUIC stream, then yeah caching is a problem. You would have to cache based on group+stream tuple. However I doubt you actually want this behavior, because you just reinvented RTMP. The whole point of groups is that they're independent, but this property is lost when they share the same QUIC stream. They wouldn't actually be independent for the first-mile delivery, and just like I mentioned in the last post, any first-mile congestion will reek havoc on last-mile delivery. I think you want to split each group into a separate QUIC stream: independent groups = independent delivery = independent streams This is why I want MoQ group == QUIC stream. However, I would be fine with MoQ object == QUIC stream, because like NAME brought up at the start, I can use a single object per group to get the desired behavior. But something needs to be specified, because a relay MUST NOT be able to move my objects to other streams.
When a new subscriber joins mid GOP, there are only 2 options to have any meaningful experience Start from the beginning of the current GOP to get the IDR Wait for the next GOP This goes to issue
I think we are mixing 2 different levels of mappings here. how an application maps its data to moqt object model One object as GOP or one object as encoded video frame of 33ms is entirely up to application. how aspects of moqt object model is mapped to the QUIC Once you have application data mapped to the moqt object model, then it comes down to question on how moqt objects/groups are transported via QUIC mechanisims. This issue talks about this mapping in particular. Yes application needs to know how to do 1 and 2, but the moqt transport only needs to specify 2. Also caching needs to be done at object level since it allows retrieval from cache provided track, group and object information as keys. I don't think we should normatively specify that relay shouldn't map things differently from ingest to egress. But yes a given relay implementation can choose to map it differently and it does indicate it via the transport delivery mode.
You can achieve this via delivery mode of stream per group. If a group has a single object (say GOP), it just caches one object for that GOP.
Yeah, my point is that you shouldn't be sending multiple GoPs over the same QUIC stream, otherwise it's not possible to serve the latest GoP during congestion. Using a single stream for multiple GoPs is only optimal when there's zero congestion, in which case just use TCP lul. Yeah, the problem is that without 2 the application can't decide 1. The transport needs to provide properties that the application can use. Right now MoqTransport OBJECTs are closer to jumbo-datagrams because they're semi-reliable and semi-ordered. It's a huge red flag when the application can't reliably deliver deltas in order because of the object/group abstraction. At a minimum, the application needs the ability to deliver a stream of bytes over MoqTransport. We should absolutely use QUIC streams for that instead of reinventing them via an increasingly convoluted object model with flags. Caching should be performed at the stream level via byte ranges. Even with objects, you want to cache at the byte range level, as waiting to receive an entire object before caching/serving can only add latency. For example, an I-frame takes multiple round-trips to deliver (especially with packet loss), so don't want to wait to receive the entire frame before caching/serving it. Here's how a relay should work: Read chunk from upstream, (optional) write chunk to cache, forward chunk to downstream. A chunk is a stream offset+length, aka a QUIC stream frame. I think the relay MUST maintain the stream mapping. Otherwise the application MUST assume the lowest common denominator, where some dumb relay decided to send every object over separate streams in an arbitrary order, or decided to drop arbitrary objects in the middle of a group. Imagine if a HTTP relay was allowed to deliver a HTTP response body in an arbitrary order or unreliably. It would break so many assumptions and force the application to anticipate this substandard delivery, even if 99% of relays delivered the body correctly. Or it would become a defacto standard that relays MUST deliver the body reliably and in order, in which case we should have been made an official standard.
Adding to this tangent, caching based on byte offset instead of frame number is far more efficient. Let me explain why. Let's assume we send an entire 2s GoP via QUIC stream. A subscriber lost their connection half-way through a GoP and wants to request it again: Ideal: Poor: In the ideal scenario, that's one lookup into the cache table. We advance the data pointer X bytes to the start and read until the requested Y byte. This memory is mostly on the same page, and can be written to the QUIC stream in one operation. In the poor scenario, the relay has to do N cache table lookups. These are likely all in separate memory pages and there's N writes to the QUIC stream. N depends on the frame rate if every frame is a separate object.
That's exactly the TransportDeliveryMode in this issue is addressing.
Applications themseleves has a choice to send every object in its own stream. Relays may choose I am not sure how was the conclusion reached here that the entirety of the object has be to received before serving downstream though. If an object is a I-Frame/GOP, the bytes/fragments come in and are sent to caching layer as fragments for building the cache for that object and also forwarded to all the subscribers in parallel as fragments. This issue for delivery mode doesn't affect that behavior in any way.
Yeah, we both agree on that front and the disagreement is just over the implementation. Instead of N different modes toggling which messages constitute a QUIC stream, I literally just need a promise that the relay can't rearrange QUIC stream contents. Even if we do add , I still need a promise that the relay won't change it. ie. if an application creates a QUIC stream with only object 1 and object 2 on it, then the relay MUST also transmit a QUIC stream with only object 1 and object 2 on it, and in that order. If a relay is allowed to split them into separate streams or insert object 3 in the middle, then it completely breaks many applications. With that property in place, I do think the stream header could deduplicate some OBJECT properties, but that's an optimization and not actually required. It just seems like a bug to send two different objects with different priorities or groups over the same QUIC stream. Relays should absolutely not be able to choose; it can only create problems. The application made the decision to deliver objects over separate QUIC streams for a very explicit reason. The relay lacks any information about the application by design so if it changes the delivery mode, it's either making a completely uninformed and likely detrimental decision, or it's applying arbitrary business logic. That was mostly from our old conversations around if objects are atomic. But I absolutely agree, relays should cache/forward streams.
I just wanted to clarify though , caching is done for objects , however relay need not wait for the full object to arrive before forwarding to the subscribers
There's no distinction. If you can serve partial objects to existing subscribers (forwarding), then you can serve those partial objects to new subscribers too (caching). In theory you could differentiate between existing and new subscribers, but that gets very complicated to implement and is detrimental when objects are not atomic (ex. GoP). You might be talking about addressability, ie. you can't request a byte range within object. But the object cache itself is absolutely broken into a dynamic list of byte chunks.
A object is the addressable entity from the moqt application perspective. How it gets delivered ( as chunks or fragments or few number of bytes) is lower layer transport level decision which is many times driven by MTu and other factors. Moqt applications ask for an object and they get it delivered. We should focus on whats the application model here
At IETF118 there was support to add explicit indicator for object model to transport mapping. PR captures 2 proposals that were discussed during Boston interim. please review and if there are suggestions o a totally different way to achieve the same results, please propose it here.
One thing missing from the discussion is ordering/reliability. The entire point of sending OBJECTs over specific streams is to utilize those properties so the application can decode in order. It's pointless to have stream mapping if OBJECTs can be reordered/dropped on those streams. Anyway, here's my proposals to round out the list: Proposal 3 (implicit): Send objects in the same manner as they were received. Proposal 4 (implicit==explicit): Same as proposal 3 but add a field to OBJECT. But like I alluded to at the start, you need more than the Stream ID to be fully explicit. You also need signals to reproduce the ordering, gaps, and stream end. And all of a sudden you're reimplemented the QUIC STREAM frame. I'm a massive fan of implicit stream mapping. I don't think there's any other option.
NAME - I think it would be good to write down in the PR the details on implicit and cover how this works when a client switched from networks or there is an error and how it works with cached data. It may be all of this is easy but I want to get my head fully around it in trying to sort this out.
Do you want me to push directly to that PR? QUIC automatically handles network migration so you're talking about a hard disconnect. Something like going through a tunnel and triggering the idle timeout (ex. 10s). The easy answer is that all streams are reset on connection loss. An application that uses long-lived streams will need to support multiple streams if they want to support reconnects. For example, if the catalog stream is reset due to connection loss, the new publisher can make a new stream with group += 1. You can still keep the ANNOUNCE/SUBSCRIBE alive, but any streams from the old publisher are reset. Attempting to resume a stream on a new connection is difficult to impossible, regardless of implicit versus explicit. The problem like I mentioned is gaps; if you're using a stream then the decoder expects objects to arrive in a specific order. The old connection would have to use QUIC ACKs to guess which streams/objects were received just in case it crashes. However, ACKs doen't actually mean that the OBJECTs were actually flushed to the application. You could guess but we would likely have to add application ACKs (ex. OBJECT_OK) primarily for this feature. You would also need the relay to throw out duplicate objects because it may have already received an object but wasn't able to ACK it. Otherwise sending the same OBJECT twice over a stream is likely to break a decoder. It's a whole mess.
Individual Comment: I have another proposal, which is is along the lines of NAME 's idea from . Combine the Proposal 2 Mode from the OBJECT wire format with the message type, and use a stream header to avoid repeating the same or potentially mismatched information. eg: StreamPerObject: Message = (same as OBJECT today -- eg no payload length) Datagram: The same as but with no serialized type field (or add one if that's too "implicit") Multi Object Streams start with a stream header message and the fields that are constant across all objects on that stream. StreamPerGroup: With similar constructs for StreamPerTrack and StreamPerPriority. The SHORTOBJECT* types are not serialized on the wire, it's inferred from the stream header. I will note this adds more flexibility than Proposal 2, eg a publisher could mix all 5 modes of object delivery in single track, though we could further restrict it if we wanted to. I wrote text for this proposal but wanted to pitch the idea here first.
Hmm, so there's some nuances to that proposal NAME Can be reordered and/or dropped? Basically, what happens when sequence=3 has a higher sendOrder (lower priority) than sequence=4?
I think we need to take a step back and really ask ourselves what properties do we want from the transport. I'm worried that we're throwing solutions at the wall without analyzing the problem. An application has live media that it needs to break into pieces. Mostly avoiding existing terminology for the moment, let's say we break a track into independent/streamable fragments which are then broken into chunks. Each fragment can have different modes: unreliable: chunks MAY be dropped unordered: chunks MAY arrive out of order framed: chunks have boundaries As a thought experiment, let's consider how an application would want to send a JSON catalog with (JSON) delta updates: independent: New catalogs don't depend on old catalogs. reliable: Delta updates can't be skipped. ordered: Delta updates need to be applied in order. framed: You can't concat two JSON objects together (without NDJSON). Each catalog would be a reliable/ordered/framed fragment in this scheme. The first chunk is the base JSON and each subsequent chunk is a JSON delta. Here's some possible use-cases for each type of fragment: Note that sending a single chunk per fragment is the same for all modes. This is what RUSH currently does and it uses a reassembly buffer to reorder. However, it would be neat if MoqTransport could perform that reordering via unreliable/ordered delivery, so the decoder could read in order by skip over any holes (ex. drop b-frames during congestion).
Okay cool, but this concept is kind of useless unless we can actually implement it. Here's what is possible in draft-01: The rest are not guaranteed due to the lack of a stream mapping. A relay can decide to reorder/drop/move OBJECTs on a whim. The only way to achieve guaranteed behavior is to use a single, unsized object per stream so the relay can't coalesce them. I think we all agree that this is a problem. Let's start with the easy proposal (3). Implicit mapping means the relay is not allowed to change the contents of a QUIC stream and all OBJECTs are delivered in the same position. You can now send multiple OBJECTs on the same stream and have them reliably delivered (ex. catalog). But if you to do more complex stuff, like unreliable/ordered (ex. lossy frames in decode order) then you still need to perform reassembly in the application. The problem with Alan's proposal is that it's unclear if chunks are reliable/ordered. If they are, then you don't need or even per . If they're not, then it's ambiguous. But what about taking that proposal a step further and explicitly include the reliable/ordered flags in the . There would always be a stream per fragment with at least one chunk. This would allow: The giant caveat here is that any dropping/reordering within a stream can only be performed at the MoqTransport layer. All chunks are reliable/ordered via QUIC streams, but the relay MAY drop chunks if there's sufficient back-pressure (like TCP low-watermark). However if latency is important, then you shouldn't perform reliable/ordered delivery in QUIC and then pretend like it unreliable/unordered in the MoQ layer. This is actually one of my complaints with SCTP; it gives the application the ability to specify unordered/unreliable delivery on a per-message basis but it's a facade. I don't think we should include reliable/unordered or unreliable/ordered messages on a QUIC stream because it's a foot gun. QUIC just doesn't work like that. Despite our object model, OBJECTs are not actually the unreliable/unordered unit, it's actually a QUIC stream and we have to remember that.
Individual Comment: I think it's interesting to consider where reorderable and droppable properties fit in the object model, but I think we can tackle the problems separately and make incremental progress. My proposal takes the draft-01 "implicit" signal and makes it explicit and more compact, without changing anything else.
unfortunately draft-01 as-is implemented wasn't implicit or explicit as shown cleary during interop.
Sure, my point is that the implicit signal exists whether you read/use it or not, and this explicit scheme conveys equivalent information.
in this case, I wonder if flexibility will come bit us. I need to think a bit more though ..
I feel we are over complicating the issue at hand by mixing different things. Let me share my thinking and see if I messed it up Every object carries an indicator for priority(sendOrder). Relay application is the one that uses this field to make forward or drop decisions and not the transport mapping and hence not scoped to this issue IIUCProposal 2 (and even Proposal 1) carries a 3 bit value in every object that indicates how a object needs to be transported. When the same object is replayed from cache the same rule applies since the object header has the necessary indicator. This mapping is what this issue is all about. Applications know what treatment they need from the transport and they need a way to indicate their intention to the moq layer and this issue is about helping build that abstraction. For applications who don't know what to do , the draft needs to specify the reasonable default.
Discussed offline with Suhas: In Proposal 3, every Object has the forwarding preference, but in some modes it is "compressed" on the wire by including a stream header. From an API perspective, publishers would publish objects with a mode and subscribers would receive objects with a mode, like Proposal 2.
I've thought about this for a bit, and I think I'm also a fan of "explicit-implicit" design. What I mean by this is that we formally define a way to describe the object placement (that is used in the APIs and can be serialized into the cache), but we don't actually send most of it on the wire since it's redundant (a "compression" of sorts).
Oh, and to be clear, I think Christian and Alan's stream header is absolutely the correct direction. My rambling is only because the presence of implies that objects on a stream are not reliable/ordered. Controversial take: For all of these stream mapping proposals, OBJECTs on a stream MUST be delivered in reliably and in order. Why? Well the entire point of mapping objects to streams is to get these properties. If a relay is allowed to arbitrarily reoreder or drop OBJECTs within a stream, then we're right back to where we started without stream mapping. For example, we want to produce a catalog with an OBJECT for each delta. These objects are placed on the same stream so the decoder doesn't need to implement a reassembly buffer or detect gaps. If we allow a relay to drop or reorder objects on the same stream for whatever reason, then this is possible: For anybody who implemented moq-clock this week, I'm sure this looks familiar. It's what you get with a stream per OBJECT, stream mapping actually served no purpose. I can live with flag indicating that objects on a stream are potentially unreliable/unordered, but I do caution that QUIC streams are reliable/ordered by nature. If an application wants unreliable or unordered OBJECTs within a stream, it's possible but head-of-line blocking will limit the upside. These objects really should be sent over a dedicated stream or datagram instead.
NAME if multiple objects are sent on a single stream, they are by definition delivered reliably and in order. Why do you think that they would not be?
OBJECTs on a stream are reliable/ordered over QUIC, but do we agree that they're reliable/ordered at rest too? Basically a relay MUST transmit OBJECTs in the same manner as they arrived.
Individual Comment: The different send order on the same stream is covered in issue , and I don't think we need to resolve that before we address the basic transport mapping.
Sorry for the earlier outburst. Alan's proposal is definitely in the right direction and is starting to align with my vision. I think I just need an outlet to write that vision down because I'm getting overwhelmed by leaving piecewise opinions on Github. I think we should: delete , unless somebody wants it. delete , moving priority to the stream header instead. add , but we should discuss datagrams more in general.Individual commentsOverall I like the proposal the explicitly map objects to quic streams. I left few comments


Currently, VarInts have two lengths: In the parameter length. In the first two bits of the parameter value. The decision in was that the two MUST match, which is a good solution without a full reform. However: Ignoring seems like the wrong play. For example, we know the role parameter (0x0) is a VarInt in the current draft. However this text says I MUST ignore any errors when parsing this varint and pretend like the parameter was never sent, which is both unenforceable and a major foot-gun (silently ignoring parameters). I think we close the connection with an error if there's an error parsing a known parameter.
If a parameter is unknown, then the parser doesn't know if the value is a varint and therefore can't check the length match. I thought it was better to have consistent behavior than not, because it's easier to write and easier to reason about. But if this has actual downstream consequences, I could be convinced otherwise.
I don't know what I was smoking when I came up with this title. My concern is that when a known parameter has the wrong length, the text says the decoder MUST ignore it. But it likely has the wrong length due to a bug, so this will cause unintended (non-fatal) behavior.
I understand what you mean. My point was that an unknown parameter has an inevitable result, and trying to make the processing for known parameters the same had some advantages. We'll discuss today.
Conclusion was to make it an error instead of skipping over the field if there's a length mismatch.
LG, though we can bikeshed about the error code name.


This was previously part of . It allows an object to be sent in a datagram f it fits, and put on a stream if it doesn't.
Linking some feedback raised in the original PR that has not yet been addressed: URL URL
Looks like a great starting point to me.