Issue 11035: Must send GAP when filtering on writer-side (ddsi-rtps-rtf) Source: Real-Time Innovations (Mr. Kenneth Brophy, ken(at)rti.com) Nature: Uncategorized Issue Severity: Summary: Title: Must send GAP when filtering on writer-side regardless of reliability QoS setting Source: Real-Time Innovations, Inc. (Ken Brophy, ken@rti.com) Summary: This issue is not currently in a state to be resolved. What follows are various thoughts on the issue and possible solutions to be discussed. This issue will not be resolved in time for the finalization task force and is included here to be documented for the revision task force. Discussion topic: what are DDS semantics for combining filtering with the deadline QoS? Should the deadline be triggered when all samples during the deadline period were filtered out? That is, does deadline require at least one sample to arrive every deadline period seconds that passed the filter? Or is the deadline satisfied when any sample arrives within the period, whether filtered out or not? If deadline only applies to samples that pass the filter, RTPS needs no changes, simply use the GAP subMessage to avoid incorrect onDataLost callbacks. If not, we run into a problem when using keyed DataWriters and finite deadlines. As the deadline applies on a per instance basis, the Reader expects at least one update for every instance, even when none of the updates for a particular instance pass the filter. A GAP message does not indicate for which instance an update was filtered out, so it cannot be used by the Reader to verify the deadline constraint. Instead, we should consider using an empty DATA message instead, possibly with a flag that states the update did not pass any filter. This would also be useful to add a new instance state NOT_ALIVE_FILTERED or so later on to the DDS spec. Another possibility would be to add a list of KeyHashes to the GAP message. SO that a GAP that is caused by a CFT actually encodes the instances that are being gapped. This would not cause incorrect firings of the DEADLINE and as a result would maintain ownership of instances even if they are filtered out… There are two ways to do this. Either we separate GAPs that correspond to filtered samples from those that correspond to irrelevant samples. So in effect we have two kinds of GAP messages Or we list explicitly the sequence number of each filtered message along with its KeyHash. Not clear what would be easier implementationwise. The samples that have been filtered are still on the writer so it appears that either implementation would work. Option (1) would save putting the sequence number with each KeyHash. This can be 4 or 8 bytes per instance, depending on whether we put the sequence number as is, or we encode it as an increment Option (2) would cause additional GAP submessages to be sent which is an overhead of 28 bytes. Not clear what is less costly… Also, if we use Option(1), then the messages that represent real GAPs can be sent via multicast; but this is only likely to occur when late joiners appear as normally there would be no "irrelevant" gaps if data is published immediately. Moreover we can in practice still do this and separate the GAP messages that represent real GAPs from the ones that don't. Option 1 does not force us to combine, just provides the means to do so… Option (2) has the problem that in certain edge cases the overhead is significant. For example if each we have a irrelevant-sample GAP followed by a filtered sample, followed by an irrelevant sample gap, etc. then we end up sending one GAP message per filtered sample with is 28 bytes of overhead per filtered sample versus a single GAP with 4 bytes of overhead per filtered sample. Also the processing is much more efficient as each GAP message is dispatched separately up the receiver's stack. For this reason it appears that Option 1 is more flexible, and the overhead is more stable. Opt Proposal(s): Always send GAPs for filtered-out messages (both in the BestEffort and in the RELIABLE) cases If the type is Keyed, then the GAP also includes at the end a sequence of : struct FilteredSampleDesc { long gapStartOffset; KeyHashPrefix keyHashPrefix; KeyHashSuffic keyHashSuffix; }; The GAP message gets two additional flags: KeyHashFlag indicates the presence of the KeyHashPrefix FilteredSamplesFlag Indicates the presence of the sequence< FilteredSampleDesc> An issue needs to be filed against the DDS spec to clarify: (a) Whether the deadline as specified by a DataReader should apply to the samples that pass the DataReader filter or to the samples sent by any writer? (b) Whether a new instance_state ALIVE_FILTERED should be added such that the DataReader can determine that a sample was filtered and potentially take action on that. (c) Whether an API or a QoS should be added to the DataReader to allow the DataReader to remove the instance information for instances with state ALIVE_FILTERED after all samples are taken. This allows resources to be conserved in the case where once filtered we expect the instance to remain filtered and also allows a reader to be notified if the instance becomes unfiltered. (d) Weather to add a filtered_generation_count that the instance has becomed ALIVE after being in the ALIVE_FILTERED Resolution: T4 should include code and description that states that when the sample is not relevant, send a GAP…same for the stateful best effort writer. Revised Text: Resolution: Revised Text: Actions taken: May 23, 2007: received issue August 1, 2007: Deferred Discussion: Proposal(s): Always send GAPs for filtered-out messages (both in the BestEffort and in the RELIABLE) cases If the type is Keyed, then the GAP also includes at the end a sequence of : struct FilteredSampleDesc { long gapStartOffset; KeyHashPrefix keyHashPrefix; KeyHashSuffic keyHashSuffix; }; The GAP message gets two additional flags: KeyHashFlag indicates the presence of the KeyHashPrefix FilteredSamplesFlag Indicates the presence of the sequence< FilteredSampleDesc> An issue needs to be filed against the DDS spec to clarify: (a) Whether the deadline as specified by a DataReader should apply to the samples that pass the DataReader filter or to the samples sent by any writer? (b) Whether a new instance_state ALIVE_FILTERED should be added such that the DataReader can determine that a sample was filtered and potentially take action on that. (c) Whether an API or a QoS should be added to the DataReader to allow the DataReader to remove the instance information for instances with state ALIVE_FILTERED after all samples are taken. This allows resources to be conserved in the case where once filtered we expect the instance to remain filtered and also allows a reader to be notified if the instance becomes unfiltered. (d) Weather to add a filtered_generation_count that the instance has becomed ALIVE after being in the ALIVE_FILTERED Resolution: T4 should include code and description that states that when the sample is not relevant, send a GAP…same for the stateful best effort writer. Resolution: This issue cannot be resolved until the DDS specification defines what the correct behavior is for readers that use both content-filtered topics and a deadline. The proposal is to leave this issue unresolved for now until the DDS specification clarifies the semantics of the deadline. The opinion of this RTF task force is that a content filter prevents non-matching data from arriving in the datareader and a deadline applies to a datareader. Therefore deadline only applies to samples that pass the filter. When read- and/or query conditions are used instead of content filters, data always arrives at the datareader and deadline applies to all samples. Revised Text: Proposed Disposition: Deferred End of Annotations:===== s is issue # 11035 Must send GAP when filtering on writer-side Title: Must send GAP when filtering on writer-side regardless of reliability QoS setting Source: Real-Time Innovations, Inc. (Ken Brophy, ken@rti.com) Summary: This issue is not currently in a state to be resolved. What follows are various thoughts on the issue and possible solutions to be discussed. This issue will not be resolved in time for the finalization task force and is included here to be documented for the revision task force. Discussion topic: what are DDS semantics for combining filtering with the deadline QoS? Should the deadline be triggered when all samples during the deadline period were filtered out? That is, does deadline require at least one sample to arrive every deadline period seconds that passed the filter? Or is the deadline satisfied when any sample arrives within the period, whether filtered out or not? If deadline only applies to samples that pass the filter, RTPS needs no changes, simply use the GAP subMessage to avoid incorrect onDataLost callbacks. If not, we run into a problem when using keyed DataWriters and finite deadlines. As the deadline applies on a per instance basis, the Reader expects at least one update for every instance, even when none of the updates for a particular instance pass the filter. A GAP message does not indicate for which instance an update was filtered out, so it cannot be used by the Reader to verify the deadline constraint. Instead, we should consider using an empty DATA message instead, possibly with a flag that states the update did not pass any filter. This would also be useful to add a new instance state NOT_ALIVE_FILTERED or so later on to the DDS spec. Another possibility would be to add a list of KeyHashes to the GAP message. SO that a GAP that is caused by a CFT actually encodes the instances that are being gapped. This would not cause incorrect firings of the DEADLINE and as a result would maintain ownership of instances even if they are filtered out. There are two ways to do this. Either we separate GAPs that correspond to filtered samples from those that correspond to irrelevant samples. So in effect we have two kinds of GAP messages Or we list explicitly the sequence number of each filtered message along with its KeyHash. Not clear what would be easier implementationwise. The samples that have been filtered are still on the writer so it appears that either implementation would work. Option (1) would save putting the sequence number with each KeyHash. This can be 4 or 8 bytes per instance, depending on whether we put the sequence number as is, or we encode it as an increment Option (2) would cause additional GAP submessages to be sent which is an overhead of 28 bytes. Not clear what is less costly. Also, if we use Option(1), then the messages that represent real GAPs can be sent via multicast; but this is only likely to occur when late joiners appear as normally there would be no "irrelevant" gaps if data is published immediately. Moreover we can in practice still do this and separate the GAP messages that represent real GAPs from the ones that don't. Option 1 does not force us to combine, just provides the means to do so. Option (2) has the problem that in certain edge cases the overhead is significant. For example if each we have a irrelevant-sample GAP followed by a filtered sample, followed by an irrelevant sample gap, etc. then we end up sending one GAP message per filtered sample with is 28 bytes of overhead per filtered sample versus a single GAP with 4 bytes of overhead per filtered sample. Also the processing is much more efficient as each GAP message is dispatched separately up the receiver's stack. For this reason it appears that Option 1 is more flexible, and the overhead is more stable. Opt Proposal(s): Always send GAPs for filtered-out messages (both in the BestEffort and in the RELIABLE) cases If the type is Keyed, then the GAP also includes at the end a sequence of : struct FilteredSampleDesc { long gapStartOffset; KeyHashPrefix keyHashPrefix; KeyHashSuffic keyHashSuffix; }; The GAP message gets two additional flags: KeyHashFlag indicates the presence of the KeyHashPrefix FilteredSamplesFlag Indicates the presence of the sequence< FilteredSampleDesc> An issue needs to be filed against the DDS spec to clarify: (a) Whether the deadline as specified by a DataReader should apply to the samples that pass the DataReader filter or to the samples sent by any writer? (b) Whether a new instance_state ALIVE_FILTERED should be added such that the DataReader can determine that a sample was filtered and potentially take action on that. (c) Whether an API or a QoS should be added to the DataReader to allow the DataReader to remove the instance information for instances with state ALIVE_FILTERED after all samples are taken. This allows resources to be conserved in the case where once filtered we expect the instance to remain filtered and also allows a reader to be notified if the instance becomes unfiltered. (d) Weather to add a filtered_generation_count that the instance has becomed ALIVE after being in the ALIVE_FILTERED Resolution: T4 should include code and description that states that when the sample is not relevant, send a GAP.same for the stateful best effort writer. Revised Text: OMG Issue No: 11035 R#10 Title: Must send GAP when filtering on writer-side regardless of reliability QoS setting Source: Real-Time Innovations, Inc. (Ken Brophy, ken@rti.com) Summary: What are DDS semantics for combining filtering with the deadline QoS? Should the deadline be triggered when all samples during the deadline period were filtered out? That is, does deadline require at least one sample to arrive every deadline period seconds that passed the filter? Or is the deadline satisfied when any sample arrives within the period, whether filtered out or not? If deadline only applies to samples that pass the filter, RTPS needs no changes, simply use the GAP subMessage to avoid incorrect onDataLost callbacks. If not, we run into a problem when using keyed DataWriters and finite deadlines. As the deadline applies on a per instance basis, the Reader expects at least one update for every instance, even when none of the updates for a particular instance pass the filter. A GAP message does not indicate for which instance an update was filtered out, so it cannot be used by the Reader to verify the deadline constraint. Instead, we should consider using an empty DATA message instead, possibly with a flag that states the update did not pass any filter. This would also be useful to add a new instance state NOT_ALIVE_FILTERED or so later on to the DDS spec. Another possibility would be to add a list of KeyHashes to the GAP message. So that a GAP that is caused by a CFT actually encodes the instances that are being gapped. This would not cause incorrect firings of the DEADLINE and as a result would maintain ownership of instances even if they are filtered out. There are two ways to do this. · Either we separate GAPs that correspond to filtered samples from those that correspond to irrelevant samples. So in effect we have two kinds of GAP messages · Or we list explicitly the sequence number of each filtered message along with its KeyHash. Not clear what would be easier implementation wise. The samples that have been filtered are still on the writer so it appears that either implementation would work. · Option (1) would save putting the sequence number with each KeyHash. This can be 4 or 8 bytes per instance, depending on whether we put the sequence number as is, or we encode it as an increment · Option (2) would cause additional GAP submessages to be sent which is an overhead of 28 bytes. Not clear what is less costly. Also, if we use Option(1), then the messages that represent real GAPs can be sent via multicast; but this is only likely to occur when late joiners appear as normally there would be no "irrelevant" gaps if data is published immediately. Moreover we can in practice still do this and separate the GAP messages that represent real GAPs from the ones that don't. Option 1 does not force us to combine, just provides the means to do so. Option (2) has the problem that in certain edge cases the overhead is significant. For example if each we have a irrelevant-sample GAP followed by a filtered sample, followed by an irrelevant sample gap, etc. then we end up sending one GAP message per filtered sample with is 28 bytes of overhead per filtered sample versus a single GAP with 4 bytes of overhead per filtered sample. Also the processing is much more efficient as each GAP message is dispatched separately up the receiver's stack. For this reason it appears that Option 1 is more flexible, and the overhead is more stable. A reasonable proposal would be to always send GAPs for filtered-out messages (both in the BestEffort and in the RELIABLE) cases If the type is Keyed, then the GAP also includes at the end a sequence of : struct FilteredSampleDesc { long gapStartOffset; KeyHashPrefix keyHashPrefix; KeyHashSuffix keyHashSuffix; }; The GAP message gets two additional flags: KeyHashFlag indicates the presence of the KeyHashPrefix and FilteredSamplesFlag, indicating the presence of the sequence< FilteredSampleDesc> An issue needs to be filed against the DDS spec to clarify: (a) Whether the deadline as specified by a DataReader should apply to the samples that pass the DataReader filter or to the samples sent by any writer? (b) Whether a new instance_state ALIVE_FILTERED should be added such that the DataReader can determine that a sample was filtered and potentially take action on that. (c) Whether an API or a QoS should be added to the DataReader to allow the DataReader to remove the instance information for instances with state ALIVE_FILTERED after all samples are taken. This allows resources to be conserved in the case where once filtered we expect the instance to remain filtered and also allows a reader to be notified if the instance becomes unfiltered. (d) Weather to add a filtered_generation_count that the instance has become ALIVE after being in the ALIVE_FILTERED Resolution: This issue cannot be resolved until the DDS specification defines what the correct behavior is for readers that use both content-filtered topics and a deadline. The proposal is to leave this issue unresolved for now until the DDS specification clarifies the semantics of the deadline. The opinion of this RTF task force is that a content filter prevents non-matching data from arriving in the datareader and a deadline applies to a datareader. Therefore deadline only applies to samples that pass the filter. When read- and/or query conditions are used instead of content filters, data always arrives at the datareader and deadline applies to all samples. Revised Text: Disposition: Closed NoChange X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to :content-type; bh=5Nd9EdyLaV5j+LChrmby9MGP3K153EhlrSAmATVbQp0=; b=VvSXNK3eik6SYmiJmB0Ve2hmtm/8YYIbBeIAnrRLd32OKB6qkfiwEr2LiIMAwEQFZ6 3ht/+JjcIAiRqsYpMkuxLuTL8EXS7dLe1n/w9ELEB/3k5Ig8mE3lpJ0Uj7u8CQ6ftvIT 7Hrva8v1jQLVeCzHg+b5oLBvJmL4tqDhkEfS6e60l5bL83hyRarXewv08MUL3wQxiTi/ 7xz/ZRFWaoBjH+u/iVVOKvmeSqm45EDPZTJRH/wnim2quVQZ178pWPiJcG0tzbePQOtl hgJ7gM+6LuTFmcwuyr5egidOhGYI3fxc5IwLSQ6lC/1NUUR37+50A2aVVQX3e8Ojw+pi XIsA== X-Gm-Message-State: ALoCoQmh6FI4Vr/cd3sEyhuAXneGiKdU6TSn6I6q+rhHU7v63UJZIEdhjEikIeKUwmGXkmltxpGm X-Received: by 10.194.58.19 with SMTP id m19mr137531488wjq.52.1420586714635; Tue, 06 Jan 2015 15:25:14 -0800 (PST) From: Gerardo Pardo Date: Tue, 6 Jan 2015 15:24:54 -0800 Subject: Discussion and proposal for RTPS RTF3 Issue number 11035 To: "ddsi-rtps-rtf@omg.org" X-Virus-Scanned: amavisd-new at omg.org Hi, Happy New Year to all! I hope all of you enjoyed a nice vacation. I have prepared a document summarizing the background, past discussions, and latest proposal for 11035. This is one of the complex issues that we had to defer and affects both DDS and RTPS. I uploaded the proposal to a dropbox folder I shared with the implementors in the RTF. The direct link in case you do not get the invite is: https://www.dropbox.com/s/b0b2lausbcozmtz/dds_interop_rtf3_issue_11035.docx?dl=0 I think this is an important issue because it affects the correctness of DDS systems when using content-filtered Topics. I would also like to start weekly conferences at least among those in the RTF that are implementors of the protocol. To arrange the best time for all I have prepared a poll. Please follow this link and enter the times that you would be available. I only specified 2 weeks but please fill the times that are best also for the weeks that follow: http://doodle.com/38s5wk6rrw677w2g Over the next few days I will prepare proposals for some of the other issues but I figured I would get started with this one as it will require some time to review... Regards, Gerardo -- Gerardo Pardo-Castellote, Ph.D. CTO - Real-Time Innovations, Inc. RTI - Your Systems working as One