Issue 8969: Allowing Mutual Recursion for IDL Structures (corba-rtf) Source: Zuehlke Engineering (Mr. Frank Pilhofer, fpilhofer2008(at)gmail.com) Nature: Uncategorized Issue Severity: Summary: CORBA 2.4 introduced forward declarations for IDL structures and unions in support of recursive structures, deprecating the prior practice of anonymous types. Also allowed were sequences of forward-declared structures ("incomplete types"), which could then be used as members in defining the structure. Currently, it is only allowed to use incomplete types in the actual definition of the type itself. As an example in section 3.11.2.3 demonstrates, this does allow indirect recursion -- but only if the incomplete types are nested, as in [first example] struct Foo; typedef sequence<Foo> FooSeq; struct Foo { struct Bar { FooSeq fs; } b; }; Specifically not allowed -- and this is the point of this issue -- is the seemingly more intuitive definition of [second example] struct Foo; typedef sequence<Foo> FooSeq; struct Bar { FooSeq fs; }; struct Foo { Bar b; }; Currently, the spec says that, "sequence members that are recursive must refer to an incomplete type currently under definition," and thus Bar is not allowed to use FooSeq as a member. However, the second example is, in effect, no different than the first. In the first example, "Foo::Bar" is a well-defined stand-alone type that can be used elsewhere (e.g., as a structure member or operation parameter). If a developer intends to use both structures, the second example makes this much clearer, as it syntactically elevates "Foo::Bar" from a mere sub-type to an "independent" structure. Therefore, I would like to change the current wording of section 3.11.2.3 to allow the second example. A proposed update is below. This issue is all the more urgent because another available specification, the "Deployment and Configuration of Component-based Distributed Applications," depends on it, by using two IDL structures that mutually and indirectly recurse, effectively using [third example] struct Package; typedef sequence<Package> PackageSeq; struct Assembly; typedef sequence<Assembly> AssemblySeq; struct Package { AssemblySeq as; }; struct Assembly { PackageSeq ps; }; In reality, the IDL in question is a bit more complicated, using some intermediate structures, which makes rewriting the IDL without this mutual recursion impractical and non-intuitive -- also because both "Package" and "Assembly" are meant to be potentially top-level, stand-alone items. Some might argue that the IDL restriction existed before the "Deployment" specification was adopted, and that CORBA should not be changed just because some later spec willingly (or rather, naively) used buggy IDL. So let me make some more arguments in favor of my request. First, as explained above, IDL already allows for indirect recursion. It just requires nesting. Second, defining structures as a "side-effect" of a member declaration is ugly, only marginally better than anonymous types. Allowing the type definition of a member to stand by itself is, in my opinion, much cleaner. Third, indirect recursion between non-nested types is no more difficult to implement in an ORB than indirect recursion between nested types. In fact, some existing ORB products already have no problem with indirect recursion, and are able to compile the IDL from the third example, resulting in correct code. The code works fine with Mico, TAO, JacORB and Combat, all of which apparently neglect to implement the check that "sequence members that are recursive must refer to an incomplete type currently under definition." OmniORB does issue a diagnostic, but simply removing the check, and making another trivial change to its IDL compiler, results in correct C++ code. Four, the existing IDL syntax, TypeCodes, CDR marshalling rules, and Interface Repository all allow indirect recursion to exist. In fact, it is already possible to create the above data types using the Interface Repository and TypeCode interfaces -- as well as to create instances using DynamicAny, and to marshal them. With this background, I suggest to remove the statement that prevents indirect recursion between non-nested structures and unions. Proposed resolution: In section 3.12.2.3, change paragraphs (counting each IDL code example as a single paragraph) 10 to 12 (page 3-42) from If a recursive structure or union member is used, sequence members that are recursive must refer to an incomplete type currently under definition. For example struct Foo; typedef sequence<Foo> FooSeq; struct Bar { long value; FooSeq chain; // Illegal, Foo is not an enclosing }; // struct or union. Compilers shall issue a diagnostic if this rule is violated. to If a sequence member of a structure or union refers to an incomplete type, the structure or union itself remains incomplete until the member's definition is completed. For example struct Foo; typedef sequence<Foo> FooSeq; struct Bar { long value; FooSeq chain; // Use of incomplete type }; // Bar itself remains incomplete struct Foo { // ... }; // Foo and Bar are complete Thank you for listening. Also thanks to Jeff Parsons and Boris Kolpakov from Vanderbilt University for researching this issue. We, the submitters of the "Deployment" specification, genuinely believe that indirect recursion is useful, and its lack (and having to work around) would take considerable value from the specification. I am uncomfortable arguing to change another spec to fix ours. But one spec has to change, and I believe that indirect recursion is a useful feature that already (unwillingly) exists in many ORBs, that it is no more problematic to implement than the existing means of recursion, and that the resulting data types are already valid when obtained from the TypeCode or Interface Repository interfaces. Considering the conflict of available specifications, I am tempted to flag this issue as urgent. Andrew, is that even possible, given that there is no active Core RTF? Resolution: Revised Text: Actions taken: August 17, 2005: received issue April 11, 2012: Deferred Discussion: End of Annotations:===== ubject: Allowing Mutual Recursion for IDL Structures (Urgent Issue?) Date: Wed, 17 Aug 2005 14:58:27 -0400 Thread-Topic: Allowing Mutual Recursion for IDL Structures (Urgent Issue?) Thread-Index: AcWjXaW6DOnsmS9qQsaPKSBvX5Ki4Q== From: "Pilhofer, Frank" To: Cc: , , "Jeff Parsons" , "Douglas Schmidt" , "Andrew Watson" X-MIME-Autoconverted: from quoted-printable to 8bit by amethyst.omg.org id j7HJFGhh018317 This is a new issue for the Core RTF. As noted at the conclusion of this mail, I am tempted to make this issue urgent. Let me explain some background before getting to the point. CORBA 2.4 introduced forward declarations for IDL structures and unions in support of recursive structures, deprecating the prior practice of anonymous types. Also allowed were sequences of forward-declared structures ("incomplete types"), which could then be used as members in defining the structure. Currently, it is only allowed to use incomplete types in the actual definition of the type itself. As an example in section 3.11.2.3 demonstrates, this does allow indirect recursion -- but only if the incomplete types are nested, as in [first example] struct Foo; typedef sequence FooSeq; struct Foo { struct Bar { FooSeq fs; } b; }; Specifically not allowed -- and this is the point of this issue -- is the seemingly more intuitive definition of [second example] struct Foo; typedef sequence FooSeq; struct Bar { FooSeq fs; }; struct Foo { Bar b; }; Currently, the spec says that, "sequence members that are recursive must refer to an incomplete type currently under definition," and thus Bar is not allowed to use FooSeq as a member. However, the second example is, in effect, no different than the first. In the first example, "Foo::Bar" is a well-defined stand-alone type that can be used elsewhere (e.g., as a structure member or operation parameter). If a developer intends to use both structures, the second example makes this much clearer, as it syntactically elevates "Foo::Bar" from a mere sub-type to an "independent" structure. Therefore, I would like to change the current wording of section 3.11.2.3 to allow the second example. A proposed update is below. This issue is all the more urgent because another available specification, the "Deployment and Configuration of Component-based Distributed Applications," depends on it, by using two IDL structures that mutually and indirectly recurse, effectively using [third example] struct Package; typedef sequence PackageSeq; struct Assembly; typedef sequence AssemblySeq; struct Package { AssemblySeq as; }; struct Assembly { PackageSeq ps; }; In reality, the IDL in question is a bit more complicated, using some intermediate structures, which makes rewriting the IDL without this mutual recursion impractical and non-intuitive -- also because both "Package" and "Assembly" are meant to be potentially top-level, stand-alone items. Some might argue that the IDL restriction existed before the "Deployment" specification was adopted, and that CORBA should not be changed just because some later spec willingly (or rather, naively) used buggy IDL. So let me make some more arguments in favor of my request. First, as explained above, IDL already allows for indirect recursion. It just requires nesting. Second, defining structures as a "side-effect" of a member declaration is ugly, only marginally better than anonymous types. Allowing the type definition of a member to stand by itself is, in my opinion, much cleaner. Third, indirect recursion between non-nested types is no more difficult to implement in an ORB than indirect recursion between nested types. In fact, some existing ORB products already have no problem with indirect recursion, and are able to compile the IDL from the third example, resulting in correct code. The code works fine with Mico, TAO, JacORB and Combat, all of which apparently neglect to implement the check that "sequence members that are recursive must refer to an incomplete type currently under definition." OmniORB does issue a diagnostic, but simply removing the check, and making another trivial change to its IDL compiler, results in correct C++ code. Four, the existing IDL syntax, TypeCodes, CDR marshalling rules, and Interface Repository all allow indirect recursion to exist. In fact, it is already possible to create the above data types using the Interface Repository and TypeCode interfaces -- as well as to create instances using DynamicAny, and to marshal them. With this background, I suggest to remove the statement that prevents indirect recursion between non-nested structures and unions. Proposed resolution: In section 3.12.2.3, change paragraphs (counting each IDL code example as a single paragraph) 10 to 12 (page 3-42) from If a recursive structure or union member is used, sequence members that are recursive must refer to an incomplete type currently under definition. For example struct Foo; typedef sequence FooSeq; struct Bar { long value; FooSeq chain; // Illegal, Foo is not an enclosing }; // struct or union. Compilers shall issue a diagnostic if this rule is violated. to If a sequence member of a structure or union refers to an incomplete type, the structure or union itself remains incomplete until the member's definition is completed. For example struct Foo; typedef sequence FooSeq; struct Bar { long value; FooSeq chain; // Use of incomplete type }; // Bar itself remains incomplete struct Foo { // ... }; // Foo and Bar are complete Thank you for listening. Also thanks to Jeff Parsons and Boris Kolpakov from Vanderbilt University for researching this issue. We, the submitters of the "Deployment" specification, genuinely believe that indirect recursion is useful, and its lack (and having to work around) would take considerable value from the specification. I am uncomfortable arguing to change another spec to fix ours. But one spec has to change, and I believe that indirect recursion is a useful feature that already (unwillingly) exists in many ORBs, that it is no more problematic to implement than the existing means of recursion, and that the resulting data types are already valid when obtained from the TypeCode or Interface Repository interfaces. Considering the conflict of available specifications, I am tempted to flag this issue as urgent. Andrew, is that even possible, given that there is no active Core RTF? Cc: "Pilhofer, Frank" , issues@omg.org, corba-rtf@omg.org, deployment-rtf@omg.org, Jeff Parsons , Douglas Schmidt , Andrew Watson From: Stephen Vinoski Subject: Re: Allowing Mutual Recursion for IDL Structures (Urgent Issue?) Date: Wed, 17 Aug 2005 16:39:12 -0400 To: Jishnu Mukerji X-Mailer: Apple Mail (2.733) X-OriginalArrivalTime: 17 Aug 2005 20:39:14.0759 (UTC) FILETIME=[BB355570:01C5A36B] IONA has implemented this part of the spec, but we would not find it difficult to make the change that Frank suggests. Frank could use valuetypes to solve the problem, but I don't think I would wish valuetypes on anyone who's trying to get real work done, since not all ORBs properly or fully implement them. Either way, the spec ought to be fixed as Frank suggests since I can't think of any reason that structs should be restricted in this manner while valuetypes aren't. --steve On Aug 17, 2005, at 4:08 PM, Jishnu Mukerji wrote: Pilhofer, Frank wrote: Considering the conflict of available specifications, I am tempted to flag this issue as urgent. Andrew, is that even possible, given that there is no active Core RTF? Sure. In the absence of an RTF the AB gets to do the recommending. BTW, the Urgent process was setup specifically for resolving "interpretation" issues, and strictly speaking there is no ambiguity in interpretation of the CORBA Core specification involved in this case. So technically this would be a bit of a stretch from that angle. But for getting around that, one could argue the "conflict" issue, as you have done. It would be good to hear if any of the CORBA implementers in good faith have implemented the specification as it exists today and would find it difficult to make this sort of a change in their product. They would potentially have a good case to petition against this proposed change in CORBA Core. Let me hasten to add that I do not have a firm position on any side of this issue at this point. We should carefully look at it from various relevant angles and come to an informed conclusion. Jishnu. Date: Wed, 17 Aug 2005 16:08:50 -0400 From: Jishnu Mukerji Organization: HP Software AMOC/ATG User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.3) Gecko/20040910 X-Accept-Language: en-us, en To: "Pilhofer, Frank" Cc: issues@omg.org, corba-rtf@omg.org, deployment-rtf@omg.org, Jeff Parsons , Douglas Schmidt , Andrew Watson Subject: Re: Allowing Mutual Recursion for IDL Structures (Urgent Issue?) Pilhofer, Frank wrote: Considering the conflict of available specifications, I am tempted to flag this issue as urgent. Andrew, is that even possible, given that there is no active Core RTF? Sure. In the absence of an RTF the AB gets to do the recommending. BTW, the Urgent process was setup specifically for resolving "interpretation" issues, and strictly speaking there is no ambiguity in interpretation of the CORBA Core specification involved in this case. So technically this would be a bit of a stretch from that angle. But for getting around that, one could argue the "conflict" issue, as you have done. It would be good to hear if any of the CORBA implementers in good faith have implemented the specification as it exists today and would find it difficult to make this sort of a change in their product. They would potentially have a good case to petition against this proposed change in CORBA Core. Let me hasten to add that I do not have a firm position on any side of this issue at this point. We should carefully look at it from various relevant angles and come to an informed conclusion. Jishnu. smime1.p7s X-Sender: andrew@192.67.184.64 (Unverified) Date: Thu, 18 Aug 2005 12:49:13 +0100 To: "Pilhofer, Frank" From: Andrew Watson Subject: Re: Allowing Mutual Recursion for IDL Structures (Urgent Issue?) Cc: , , , "Jeff Parsons" , "Douglas Schmidt" Frank, You wrote: > This is a new issue for the Core RTF. As noted at the > conclusion of this mail, I am tempted to make this issue > urgent. > > Let me explain some background before getting to the point. > Thank you for listening. Also thanks to Jeff Parsons > and Boris Kolpakov from Vanderbilt University for > researching this issue. Thanks for a very complete explanation of the issue. > We, the submitters of the "Deployment" specification, > genuinely believe that indirect recursion is useful, > and its lack (and having to work around) would take > considerable value from the specification. > > I am uncomfortable arguing to change another spec > to fix ours. But one spec has to change, and I believe > that indirect recursion is a useful feature that already > (unwillingly) exists in many ORBs, that it is no more > problematic to implement than the existing means of > recursion, and that the resulting data types are already > valid when obtained from the TypeCode or Interface > Repository interfaces. > > Considering the conflict of available specifications, > I am tempted to flag this issue as urgent. Andrew, is > that even possible, given that there is no active Core > RTF? Yes, as Jishnu points out, Urgent Issues for specs with no current RTF or FTF are handed to the AB, for want of anyone better-qualified to decide their resolution. Before making a decision on whether to flag this as urgent, I'd be interested to hear from other ORB vendors. So far Iona has said "definitely an easy change for us" and Sun "probably an easy change". Anyone else? Thanks, X-Sender: andrew@192.67.184.64 Date: Fri, 2 Sep 2005 19:29:18 +0100 To: Pete Rivett , Jishnu Mukerji , Sridhar Iyengar , Julien Maisonneuve , Tom Rutt , Sumeet Malhotra , Manfred Koethe , Bob Kukura , Stephen Mellor , Victor Giddings From: Andrew Watson Subject: Issue 8969 declared Urgent Cc: Frank Pilhofer , issues@omg.org Dear AB member, At the request of Frank Pilhofer, I am declaring Issue 8969 urgent. This issue concerns Mutual Recursion for IDL Structures. Since there is no CORBA RTF at the moment, this issue will be voted on by the Architecture Board. The summary of the issue can be found here: http://www.omg.org/issues/issue8969.txt When replying to this email to discuss the issue, please don't edit the Subject line, and please keep "issues@omg.org" in the recipient list - that way Juergen can easily file the discussion at the above URL for posterity. The default resolution will be to apply the following change to the CORBA 3.0.3 specification, formal/2004-03-12. Should you vote to make a change to CORBA in response to this issue, that would theoretically create a new dot dot CORBA release (3.0.4) - but in fact we'll probably incorporate any change into CORBA 3.1. The deadline for resolving this issue is Friday 16th September (the last day of the Atlanta meeting). If an AB vote on the issue has not completed by then, I will apply this default resolution (but I'll try very hard to make sure there is an AB vote). Would anyone like to discuss the issue? If I hear no discussion by the middle of next week (Wednesday 7th September), I'll start an AB vote. --- Default Resolution Text --- In section 3.12.2.3, change paragraphs (counting each IDL code example as a single paragraph) 10 to 12 (page 3-42) from If a recursive structure or union member is used, sequence members that are recursive must refer to an incomplete type currently under definition. For example struct Foo; typedef sequence FooSeq; struct Bar { long value; FooSeq chain; // Illegal, Foo is not an enclosing }; // struct or union. Compilers shall issue a diagnostic if this rule is violated. to If a sequence member of a structure or union refers to an incomplete type, the structure or union itself remains incomplete until the member's definition is completed. For example struct Foo; typedef sequence FooSeq; struct Bar { long value; FooSeq chain; // Use of incomplete type }; // Bar itself remains incomplete struct Foo { // ... }; // Foo and Bar are complete Thanks, Andrew Subject: RE: Issue 8969 declared Urgent Date: Sun, 4 Sep 2005 17:25:52 -0700 X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Issue 8969 declared Urgent Thread-Index: AcWv7EMeKpd1hsJWQL+Q9spwxvXowgBwr82A From: "Pete Rivett" To: "Andrew Watson" , "Jishnu Mukerji" , "Sridhar Iyengar" , "Julien Maisonneuve" , "Tom Rutt" , "Sumeet Malhotra" , "Manfred Koethe" , "Bob Kukura" , "Stephen Mellor" , "Victor Giddings" Cc: "Frank Pilhofer" , X-MIME-Autoconverted: from quoted-printable to 8bit by amethyst.omg.org id j850iNhh019172 I think the resolution needs more work - unless there is a well-defined compiler processing model, and definition of 'incompleteness' and associated processing, that renders the following redundant... I'm unclear on "remains incomplete until" and the assumptions implied. a) what is the impact of this incompleteness in terms of what is not permitted/will give an error? Or it just to say that if completeness is not achieved by the end of the compilation then "Compilers shall issue a diagnostic" b) does 'until' imply a time period - in which case the duration of the above incompleteness will depend on the order of the complier processing the text of the IDL - or does it imply a span of text regardless of the order in which the compiler processes it? Pete > -----Original Message----- > From: Andrew Watson [mailto:andrew@omg.org] > Sent: Friday, September 02, 2005 7:29 PM > To: Pete Rivett; Jishnu Mukerji; Sridhar Iyengar; Julien > Maisonneuve; Tom Rutt; Sumeet Malhotra; Manfred Koethe; Bob > Kukura; Stephen Mellor; Victor Giddings > Cc: Frank Pilhofer; issues@omg.org > Subject: Issue 8969 declared Urgent > > Dear AB member, > > At the request of Frank Pilhofer, I am declaring Issue 8969 > urgent. This > issue concerns Mutual Recursion for IDL Structures. Since > there is no CORBA > RTF at the moment, this issue will be voted on by the > Architecture Board. > > The summary of the issue can be found here: > > http://www.omg.org/issues/issue8969.txt > > When replying to this email to discuss the issue, please > don't edit the > Subject line, and please keep "issues@omg.org" in the > recipient list - that > way Juergen can easily file the discussion at the above URL > for posterity. > > The default resolution will be to apply the following change > to the CORBA > 3.0.3 specification, formal/2004-03-12. Should you vote to > make a change to > CORBA in response to this issue, that would theoretically > create a new dot > dot CORBA release (3.0.4) - but in fact we'll probably incorporate any > change into CORBA 3.1. > > The deadline for resolving this issue is Friday 16th > September (the last > day of the Atlanta meeting). If an AB vote on the issue has > not completed > by then, I will apply this default resolution (but I'll try > very hard to > make sure there is an AB vote). > > Would anyone like to discuss the issue? If I hear no discussion by the > middle of next week (Wednesday 7th September), I'll start an AB vote. > > --- Default Resolution Text --- > > In section 3.12.2.3, change paragraphs (counting each IDL > code example as a single paragraph) 10 to 12 (page 3-42) > from > > If a recursive structure or union member is used, > sequence members that are recursive must refer to > an incomplete type currently under definition. For > example > > struct Foo; > typedef sequence FooSeq; > struct Bar { > long value; > FooSeq chain; // Illegal, Foo is not an enclosing > }; // struct or union. > > Compilers shall issue a diagnostic if this rule is > violated. > > to > > If a sequence member of a structure or union refers > to an incomplete type, the structure or union itself > remains incomplete until the member's definition is > completed. For example > > struct Foo; > typedef sequence FooSeq; > struct Bar { > long value; > FooSeq chain; // Use of incomplete type > }; // Bar itself remains incomplete > struct Foo { > // ... > }; // Foo and Bar are complete > > > Thanks, > > Andrew > Subject: Re: Issue 8969 declared Urgent Date: Mon, 5 Sep 2005 09:00:38 -0400 Thread-Topic: Issue 8969 declared Urgent Thread-Index: AcWv7EMeKpd1hsJWQL+Q9spwxvXowgBwr82AABoHuNA= From: "Pilhofer, Frank" To: "Pete Rivett" , "Andrew Watson" , "Jishnu Mukerji" , "Sridhar Iyengar" , "Julien Maisonneuve" , "Tom Rutt" , "Sumeet Malhotra" , "Manfred Koethe" , "Bob Kukura" , "Stephen Mellor" , "Victor Giddings" Cc: X-MIME-Autoconverted: from quoted-printable to 8bit by amethyst.omg.org id j85DKEhh023313 Pete, > > I'm unclear on "remains incomplete until" and the assumptions implied. > a) what is the impact of this incompleteness in terms of what is not > permitted/will give an error? Or it just to say that if completeness is > not achieved by the end of the compilation then "Compilers shall issue a > diagnostic" > the existing text already defines incomplete types, and covers the semantics of incompleteness. Section 3.11.2.3, paragraph 9 says, If a structure or union is forward declared, a definition of that structure or union must follow the forward declaration in the same source file. Compilers shall issue a diagnostic if this rule is violated. Paragraph 15 and 17 add, An incomplete type can only appear as the element type of a sequence definition. A sequence with incomplete element type is termed an incomplete sequence type. An incomplete sequence type can appear only as the element type of another sequence, or as the member type of a structure or union definition. > > b) does 'until' imply a time period - in which case the duration of the > above incompleteness will depend on the order of the complier processing > the text of the IDL > - or does it imply a span of text regardless of the order in which the > compiler processes it? > Section 3.11.2.3, paragraph 7, defines "until" as: [...] A structure or union type is termed incomplete until its full definition is provided; that is, until the scope of the structure or union definition is closed by a terminating "}". Frank