Issue 14485: packet identification (xtce-rtf) Source: NASA (Mr. Kevin Rice, james.k.rice-1(at)nasa.gov) Nature: Uncategorized Issue Severity: Summary: Description Ludovic FAURE 2007-10-31 17:31:13 GMT In the chapter 2.3.4.4, it is written that a packet is identified as a non-abstract container that has a base container (or a message as well). I think a packet can be a non-abstract container without base container. For example, if the description contains only one container that defines a packet. There is no base container whereas this is a packet. Resolution: Revised Text: Actions taken: September 17, 2009: received issue Discussion: Comment 1 Kevin Rice 2007-10-31 20:20:48 GMT XTCE assumes that you wish to supply identifying information with the container definitions. You don't have to do this if your system knows that certain container names are certain packet definitions, but this is a little bit outside of what XTCE supports in terms of what it considers is needed in for common usage. In XTCE the two ways you can supply this are through the inheritance mechanism, and the message mechanism. They are roughly equal at a simplistic level, although the inheritance mechanism adds another "meta" layer to XTCE: an inheritance model within the instance document. Some have proposed using IncludeCondition to supply identifying infomation to packets because this is easy to process using XSLTs, and avoids using either message which requires looking up a Ref, or inheritance which either requires at minimum a ref look up or some inheritance model implementation which is harder... Comment 2 Johannes Stamminger 2007-11-12 09:50:29 GMT I agree with the author, a packet does not need to have a BaseContainer. E.g. the XTCE instance describes the general CCSDS packet layout. This includes a secondary header with a project specific "packet id" parameter describing the user data section structure of the packet. For packets whose structure is not known at a receiver, the packet still needs to be identified as a ccsds packet in basic. At least to read the packet length field to determine the next packet's start. Here the matching packet description is the generic CCSDS description in the XTCE instance *not* being abstract and *not* having a BaseContainer defined. Comment 3 Kevin Rice 2007-11-12 15:19:51 GMT Hmmm interesting use case. Comment 4 Kevin Rice 2007-11-12 18:33:25 GMT I thought about this a bit more -- what has been described is really a case of using RestrictionCriteria w/CustomAlg to supply the "identifying" info... If you supply a container that is 'standalone' some other user will not know what to do with it, unless you supply additional information. Therefore if that information can be supplied in the XTCE file itself, it would be better ... And still match the 'model' so far defined for supplying identifying information in some form using the RestrictioCriteria (or Message)... Comment 5 Johannes Stamminger 2007-11-13 07:13:54 GMT Indeed, that was my first way of implementing (in terms of writing an xtce instance) it: <RestrictionCriteria> <CustomAlgorithm name="default" shortDescription="to be used in case no other container matches"> <AlgorithmText language="pseudo">default</AlgorithmText> </CustomAlgorithm> </RestrictionCriteria> But I do not like this as the situation is IMO very common to all applications dealing with ccsds packets (or by what way do you process an unknown CCSDS packet, is it application built-in then?). Describing it this way is very hard to describe in a recommendation and I doubt any application (besides the one written by the author of the document) will process it intentionally. Therefore I would prefer to recommend in the MB that in principle a packet is - as you said - a non-abstract being restricted by use of a BaseContainer. But there may exist one non-abstract packet without BaseContainer restriction that shall be treated as default packet ... ? Comment 6 Ludovic FAURE 2007-11-13 15:44:31 GMT I agree with Johannes about recommending the possible use of a default packet. In fact, at CNES, most of the missions use CCSDS packets defined in a generic and basic description by using existence conditions. So we will have only one non-abstract container that will be the default packet. For me, this container may not have a base container. Comment 7 Kevin Rice 2007-11-13 16:29:47 GMT You are correct really that is not a very transferable way to do things. The problem is this below alone doesn't tell any processing system ENOUGH information to process it: <Container name="APacket"> <EntryList> <ParameterRef name="APID"/> <ParameterRef name="LEN"/> <ParameterRef name="BODY"/> </EntryList> </Container> EXCEPT of course that the system is sophisticated enough to process CCSDS headers, derive the length and then go from there -- sort of a "packet sniffer" scenario. Ultimately the only reason I put something in the MB for identifying packets in the telemetry container area is because various people have asked me this very question many times in the past: "I don't see a 'packetList' in XTCE, how do I determine which containers are packet descriptions?" Soooo --- I tried to come up with a rule that makes sense... One problem is figuring not only WHICH containers are packets, but which containers are NOT packets and should just be ignored by processing software... as those containers will be part of other container definitions. Again I am taking this from the position that I could receive an XTCE file from Terma or CNES or ESA -- and with a very simple program I have written list out the all the packets in each of these files. Right now CNES describes their packets using an abstract container with a long list of includeConditions for each individual packets which include OTHER containers that have the variable portions of each individual packet. They call the principle abstract packet their "generic" packet. Terma it sounds like would describe most of their packets more individually with either Message or Restriction criteria, except there may be some which have neither ... How then does my simple software handle these two cases without KNOWING something about the files being ingested beyond XTCE? That is my basic question ... Comment 8 Johannes Stamminger 2007-11-14 07:55:22 GMT Kevin, IMHO the CNES approach is not correct (as I just wrote to 409, sorry again Ludovic). Yesterday I thought about using Messages for identifying the packets. Mostly because of the problem that more than one packet description might match a packet (as in my example the concrete packet description and at least the generic ccsds one, too). I thought about using the order of having referenced the container from a MessageSet. But with the current schema this is not possible - moreover I currently cannot imagine any usecase for the current MessageSet definition. Being able to define it explicitly like e.g. <MessageSet> <Message name="packetA"> <ContainerRef containerRef="packetADescription"/> </Message> <Message name="packetB"> <ContainerRef containerRef="packetBDescription"/> </Message> ... <DefaultMessage name="ccsds"> <ContainerRef containerRef="ccsdsDescription"/> </DefaultMessage> </MessageSet> I would have one - but this is regarding the schema ... Concerning your short example container: of course, this is sufficient - and it is needed/mandatory to process. At least if you want to process a stream of ccsds packets that may contain packets not being described in the xtce document relying on. And I can image that this is not only valid for ccsds packets. I am not the network expert but I'm quite sure that most protocols provide the length of the packet somewhere within, I do not believe that much protocols defining the packets of a fixed size (MIL-1553 might be one?). For all those protocols with varying packet's length such a default packet describing the basic structure is needed. Comment 9 Kevin Rice 2007-11-14 13:35:12 GMT I think the problem of finding "all the packets" in the XTCE description is typified by this example: <Container name="tiny"> <Entrylist> <ParameterRef name="len"> <Entrylist> </Container> <Container name="tiny2"> <Entrylist> <ParameterRef name="body"> <Entrylist> <Basecontainer name="tiny"/> </Container> The question is -- do I have ONE or TWO packets? As it stand now the MB says "Tiny2" is the packet. This is an extreme example but I think represents what is being discussed (???) So the need is to mark a specific description as "default"? In which case "tiny" would represent that case with just its "header" being defined. Comment 10 Kevin Rice 2007-11-14 13:45:54 GMT I think you've reflected your organization's needs within your XTCE documents and that's fine. Just like at some level Johannes is suggesting the need for a "default" packet in XTCE and that reflects his experience with processing CCSDS TLM. One "problem" then with XTCE is an inability to enforce semantics -- we have semantics for the various constructs and the MB tries to capture them. Perhaps the MB is not perfect in this regard as well and needs some improvement. Implementing the same semantics ultimately means to me the ability to exchange more broadly with others without everyone having to change their software. As it stands right now if you send me CNES files (which you have done) -- my software won't decompose that generic description into multiple concrete packet descriptions. BUT it would NOT be a huge effort to support it. In my mind that means you have several options: 1- ignore some portions of MB and then know that folks using your XTCE files will support your concept of the generic packet... 2- go ahead and put the concrete packet information in, and ignore the information for the purposes of your processing... Comment 11 Ludovic FAURE 2007-11-15 09:33:37 GMT I understand that the most important point is the interoperability of different softwares that use XTCE. Because we need that our input XTCE file describe a generic packet, we have a gateway that translate any input XTCE file (with inheritance, message, etc..) in the form we expects and we are able to process. I think our case is just a particular case where there is only one packet definition in the description, isn't? I think this should be handle as default behaviour where only one packet is defined in a XTCE file. The problem is more about how can we restituate the packet identification that we received from other agencies and that we dont process. I have an idea and i would like to know if your softwares are able to handle it in the rigth way: For us packet is identified by a set of fixed values (e.g if APID == 1000, Service_Type == 1 and Service_subtype ==2 then it is packet1) and that's all. All fields and containers are defined in the unique packet description with use of include conditions. So if i said packet1 inheriths from myGenericPacket (or message with reference to myGenericPacket ) with those restriction criteria, is it what you expect to manage? Comment 12 Kevin Rice 2007-11-15 12:59:14 GMT As to your last comment, yes I believe taking each includeCondition from the "generic packet" (&removing), making it into a restrictionCriteria and then subclassing the generic packet would do it, and is the same information. So right now: <container name="CNESGenericPacket" abstract="true"> <!-- short hand ... --> <EntryList> <parameterRef name="Header"/> <includeCondition name="Pkt1" Header.APID="3" Header.LEN="4"/> <includeCondition name="Pkt2" Header.APID="100" Header.LEN="87"/> </EntryList> </container> <container name="Pkt1"> <!-- entrylist stuff --> </container> <container name="Pkt1"> <!-- entrylist stuff --> </container> To this: (there are several possible variations to this approach depending on exactly HOW you'd like to organize your containers -- this is one) <container name="CNESGenericPacket" abstract="true"> <!-- short hand ... --> <EntryList> <parameterRef name="Header"/> <!-- includes are now restriction criteria below ... --> </EntryList> </container> <container name="Pkt1"> <!-- entrylist stuff --> <RestrictionCriteria baseName="CNESGenericPacket" Header.APID="3" Header.LEN="4"/> </container> <container name="Pkt1"> <!-- entrylist stuff --> <RestrictionCriteria baseName="CNESGenericPacket" Header.APID="100" Header.LEN="87"/> </container> Does that make sense? Now -- besides that it still seems we need a good way to specify a "DEFAULT" packet ... I'm thinking about it... Comment 13 Ludovic FAURE 2007-11-15 13:53:41 GMT In fact, this is not really what i expected. I will try to illustrate what i had in mind: Original example is: <container name="CNESGenericPacket" abstract="true"> <!-- short hand ... --> <EntryList> <parameterRef name="Header"/> <containerRef="Part1" includeCondition Header.APID="3" /> <containerRef="Part2" includeCondition Header.APID="4" /> <containerRef="Part3" includeCondition Header.APID >= 4 /> <containerRef="Part4" includeCondition Header.APID="10" /> <parameterRef name="Tail" includeCondition (Header.APID="3" || Header.APID="4") > </EntryList> </container> <container name="Part1"> <!-- entrylist stuff --> </container> <container name="Part2"> <!-- entrylist stuff --> </container> <container name="Part3"> <!-- entrylist stuff --> </container> <container name="Part4"> <!-- entrylist stuff --> </container> This is the generic packet definition. Now if i want to add information about a packet identification i will associate to a given packet name, a table of fixed field values. For instance, my packet1 can be defined like this, in addition to the previous defintion: <container/message name="packet1 "> <RestrictionCriteria baseName="CNESGenericPacket" Header.APID="3"> </container/message> packet1 will contain: HEADER-part1-Tail If packet2 contains: HEADER-part3-part4 It is written: <container/message name="packet2 "> <RestrictionCriteria baseName="CNESGenericPacket" Header.APID="10"> </container/message> Are you able to process those packets? I think this is the kind of structure that we will get at CNES from existent mission (packet identification can be added manually becasue we dont have yet). Comment 14 Kevin Rice 2007-11-15 13:58:41 GMT To me, yes -- it's not quite how i would do things but it allows you to keep your structures intact and add a little info... I'd like to draw up the UML but just looking at it -- technically yes, those added containers from my software's perspective would be the concrete packets, they'd extend the generic one and then of course have to run through all the include conditions to construct they contents properly... (in fact its somewhat duplicates info but that's ok) I can't think at the moment why technically this should not be processable.. End of Annotations:===== s is issue # 14485 packet identification Description Ludovic FAURE 2007-10-31 17:31:13 GMT In the chapter 2.3.4.4, it is written that a packet is identified as a non-abstract container that has a base container (or a message as well). I think a packet can be a non-abstract container without base container. For example, if the description contains only one container that defines a packet. There is no base container whereas this is a packet. Comment 1 Kevin Rice 2007-10-31 20:20:48 GMT XTCE assumes that you wish to supply identifying information with the container definitions. You don't have to do this if your system knows that certain container names are certain packet definitions, but this is a little bit outside of what XTCE supports in terms of what it considers is needed in for common usage. In XTCE the two ways you can supply this are through the inheritance mechanism, and the message mechanism. They are roughly equal at a simplistic level, although the inheritance mechanism adds another "meta" layer to XTCE: an inheritance model within the instance document. Some have proposed using IncludeCondition to supply identifying infomation to packets because this is easy to process using XSLTs, and avoids using either message which requires looking up a Ref, or inheritance which either requires at minimum a ref look up or some inheritance model implementation which is harder... Comment 2 Johannes Stamminger 2007-11-12 09:50:29 GMT I agree with the author, a packet does not need to have a BaseContainer. E.g. the XTCE instance describes the general CCSDS packet layout. This includes a secondary header with a project specific "packet id" parameter describing the user data section structure of the packet. For packets whose structure is not known at a receiver, the packet still needs to be identified as a ccsds packet in basic. At least to read the packet length field to determine the next packet's start. Here the matching packet description is the generic CCSDS description in the XTCE instance *not* being abstract and *not* having a BaseContainer defined. Comment 3 Kevin Rice 2007-11-12 15:19:51 GMT Hmmm interesting use case. Comment 4 Kevin Rice 2007-11-12 18:33:25 GMT I thought about this a bit more -- what has been described is really a case of using RestrictionCriteria w/CustomAlg to supply the "identifying" info... If you supply a container that is 'standalone' some other user will not know what to do with it, unless you supply additional information. Therefore if that information can be supplied in the XTCE file itself, it would be better ... And still match the 'model' so far defined for supplying identifying information in some form using the RestrictioCriteria (or Message)... Comment 5 Johannes Stamminger 2007-11-13 07:13:54 GMT Indeed, that was my first way of implementing (in terms of writing an xtce instance) it: default But I do not like this as the situation is IMO very common to all applications dealing with ccsds packets (or by what way do you process an unknown CCSDS packet, is it application built-in then?). Describing it this way is very hard to describe in a recommendation and I doubt any application (besides the one written by the author of the document) will process it intentionally. Therefore I would prefer to recommend in the MB that in principle a packet is - as you said - a non-abstract being restricted by use of a BaseContainer. But there may exist one non-abstract packet without BaseContainer restriction that shall be treated as default packet ... ? Comment 6 Ludovic FAURE 2007-11-13 15:44:31 GMT I agree with Johannes about recommending the possible use of a default packet. In fact, at CNES, most of the missions use CCSDS packets defined in a generic and basic description by using existence conditions. So we will have only one non-abstract container that will be the default packet. For me, this container may not have a base container. Comment 7 Kevin Rice 2007-11-13 16:29:47 GMT You are correct really that is not a very transferable way to do things. The problem is this below alone doesn't tell any processing system ENOUGH information to process it: EXCEPT of course that the system is sophisticated enough to process CCSDS headers, derive the length and then go from there -- sort of a "packet sniffer" scenario. Ultimately the only reason I put something in the MB for identifying packets in the telemetry container area is because various people have asked me this very question many times in the past: "I don't see a 'packetList' in XTCE, how do I determine which containers are packet descriptions?" Soooo --- I tried to come up with a rule that makes sense... One problem is figuring not only WHICH containers are packets, but which containers are NOT packets and should just be ignored by processing software... as those containers will be part of other container definitions. Again I am taking this from the position that I could receive an XTCE file from Terma or CNES or ESA -- and with a very simple program I have written list out the all the packets in each of these files. Right now CNES describes their packets using an abstract container with a long list of includeConditions for each individual packets which include OTHER containers that have the variable portions of each individual packet. They call the principle abstract packet their "generic" packet. Terma it sounds like would describe most of their packets more individually with either Message or Restriction criteria, except there may be some which have neither ... How then does my simple software handle these two cases without KNOWING something about the files being ingested beyond XTCE? That is my basic question ... Comment 8 Johannes Stamminger 2007-11-14 07:55:22 GMT Kevin, IMHO the CNES approach is not correct (as I just wrote to 409, sorry again Ludovic). Yesterday I thought about using Messages for identifying the packets. Mostly because of the problem that more than one packet description might match a packet (as in my example the concrete packet description and at least the generic ccsds one, too). I thought about using the order of having referenced the container from a MessageSet. But with the current schema this is not possible - moreover I currently cannot imagine any usecase for the current MessageSet definition. Being able to define it explicitly like e.g. ... I would have one - but this is regarding the schema ... Concerning your short example container: of course, this is sufficient - and it is needed/mandatory to process. At least if you want to process a stream of ccsds packets that may contain packets not being described in the xtce document relying on. And I can image that this is not only valid for ccsds packets. I am not the network expert but I'm quite sure that most protocols provide the length of the packet somewhere within, I do not believe that much protocols defining the packets of a fixed size (MIL-1553 might be one?). For all those protocols with varying packet's length such a default packet describing the basic structure is needed. Comment 9 Kevin Rice 2007-11-14 13:35:12 GMT I think the problem of finding "all the packets" in the XTCE description is typified by this example: The question is -- do I have ONE or TWO packets? As it stand now the MB says "Tiny2" is the packet. This is an extreme example but I think represents what is being discussed (???) So the need is to mark a specific description as "default"? In which case "tiny" would represent that case with just its "header" being defined. Comment 10 Kevin Rice 2007-11-14 13:45:54 GMT I think you've reflected your organization's needs within your XTCE documents and that's fine. Just like at some level Johannes is suggesting the need for a "default" packet in XTCE and that reflects his experience with processing CCSDS TLM. One "problem" then with XTCE is an inability to enforce semantics -- we have semantics for the various constructs and the MB tries to capture them. Perhaps the MB is not perfect in this regard as well and needs some improvement. Implementing the same semantics ultimately means to me the ability to exchange more broadly with others without everyone having to change their software. As it stands right now if you send me CNES files (which you have done) -- my software won't decompose that generic description into multiple concrete packet descriptions. BUT it would NOT be a huge effort to support it. In my mind that means you have several options: 1- ignore some portions of MB and then know that folks using your XTCE files will support your concept of the generic packet... 2- go ahead and put the concrete packet information in, and ignore the information for the purposes of your processing... Comment 11 Ludovic FAURE 2007-11-15 09:33:37 GMT I understand that the most important point is the interoperability of different softwares that use XTCE. Because we need that our input XTCE file describe a generic packet, we have a gateway that translate any input XTCE file (with inheritance, message, etc..) in the form we expects and we are able to process. I think our case is just a particular case where there is only one packet definition in the description, isn't? I think this should be handle as default behaviour where only one packet is defined in a XTCE file. The problem is more about how can we restituate the packet identification that we received from other agencies and that we dont process. I have an idea and i would like to know if your softwares are able to handle it in the rigth way: For us packet is identified by a set of fixed values (e.g if APID == 1000, Service_Type == 1 and Service_subtype ==2 then it is packet1) and that's all. All fields and containers are defined in the unique packet description with use of include conditions. So if i said packet1 inheriths from myGenericPacket (or message with reference to myGenericPacket ) with those restriction criteria, is it what you expect to manage? Comment 12 Kevin Rice 2007-11-15 12:59:14 GMT As to your last comment, yes I believe taking each includeCondition from the "generic packet" (&removing), making it into a restrictionCriteria and then subclassing the generic packet would do it, and is the same information. So right now: To this: (there are several possible variations to this approach depending on exactly HOW you'd like to organize your containers -- this is one) Does that make sense? Now -- besides that it still seems we need a good way to specify a "DEFAULT" packet ... I'm thinking about it... Comment 13 Ludovic FAURE 2007-11-15 13:53:41 GMT In fact, this is not really what i expected. I will try to illustrate what i had in mind: Original example is: = 4 /> This is the generic packet definition. Now if i want to add information about a packet identification i will associate to a given packet name, a table of fixed field values. For instance, my packet1 can be defined like this, in addition to the previous defintion: packet1 will contain: HEADER-part1-Tail If packet2 contains: HEADER-part3-part4 It is written: Are you able to process those packets? I think this is the kind of structure that we will get at CNES from existent mission (packet identification can be added manually becasue we dont have yet). Comment 14 Kevin Rice 2007-11-15 13:58:41 GMT To me, yes -- it's not quite how i would do things but it allows you to keep your structures intact and add a little info... I'd like to draw up the UML but just looking at it -- technically yes, those added containers from my software's perspective would be the concrete packets, they'd extend the generic one and then of course have to run through all the include conditions to construct they contents properly... (in fact its somewhat duplicates info but that's ok) I can't think at the moment why technically this should not be processable..