Issue 2155: OBV chunk boundaries (interop) Source: (, ) Nature: Uncategorized Issue Severity: Summary: Summary: Section 15.3.4.4. of ptc/98-10-11 includes the following statement wrt to ObV chunking: "The data may be split into multiple chunks at arbitrary points except within primitive CDR types, arrays of primitive types, strings, and wstrings." Unfortunately, this provides only dubious benfits, but introduces significant problems. Traditionally, CDR has been designed so that encoding engines are only responsible for marshalling CDR primitives, and remain independent of the actual constructed IDL type being marshalled. The above clause violates this rule: the CDR stream must know that the primitives being marshalled are part of an array, understand the size of the array, distinguish sequences from arrays, etc. In addition, this is impossible to implement using the Java portable streams. Resolution: Revised Text: Actions taken: October 30, 1998: received issue March 5, 1999: moved from obv to interop March 5, 1999: closed issue Discussion: End of Annotations:===== Return-Path: From: "Matthew A. Mihic" To: Cc: Subject: ObV chunk boundaries Date: Fri, 30 Oct 1998 08:28:54 -0500 X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.2106.4 Section 15.3.4.4. of ptc/98-10-11 includes the following statement wrt to ObV chunking: "The data may be split into multiple chunks at arbitrary points except within primitive CDR types, arrays of primitive types, strings, and wstrings." Unfortunately, this provides only dubious benfits, but introduces significant problems. Traditionally, CDR has been designed so that encoding engines are only responsible for marshalling CDR primitives, and remain independent of the actual constructed IDL type being marshalled. The above clause violates this rule: the CDR stream must know that the primitives being marshalled are part of an array, understand the size of the array, distinguish sequences from arrays, etc. In addition, this is impossible to implement using the Java portable streams. The streams are not informed of the IDL type being encoded, and cannot rely on arrays being marshalled using a single invocation of the write_x_array methods (or even using this method at all). Instead of this definition, I suggest that we allow chunks to end anywhere that a fragment could end. This simplifies the encoding rules (as there is no need for a marshalling engine to remember where a chunk can end vs. where a fragment can end), normalizes the definition, and becomes both possible and easy to implement using the Java portable streams. The above sentence would be rewritten to say: "The data may be split into multiple chunks in the same manner as fragments." Regards, Matt -- Matthew A. Mihic IONA Technologies, Inc. Phone: +1-617-949-4302 EMail: mmihic@iona.com Sender: jis@fpk.hp.com Date: Thu, 04 Mar 1999 17:53:44 -0500 From: Jishnu Mukerji Organization: Hewlett-Packard EIAL To: issues@omg.org, juergen@omg.org Subject: Issue 2155 Issue 2155 pertains to chapter 15 and is therefore not a core issue. Please move it to Interop. To restate the rule: Core Chapter 12, 13, 14 and 15 belong to Interop RTF. Please do not assign issues that specifically mention those chapter to the Core RTF. Thanks, Jishnu. -- Jishnu Mukerji Systems Architect Email: jis@fpk.hp.com Hewlett-Packard EIAL, Tel: +1 973 443 7528 300 Campus Drive, 2E-62, Fax: +1 973 443 7422 Florham Park, NJ 07932, USA. From: "Rutt, T E (Tom)" To: "'interop@Omg.org'" Cc: "Rutt, T E (Tom)" Subject: Need Help on Issue 2155 Reso - OBV ChunckBoundary Date: Mon, 6 Nov 2000 23:45:27 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain X-UIDL: 6QDe9VAnd9@R]d9T=md9 Please read the discussion section below. I welcome comments/alternatives for discussion. Tom Rutt Terutt@lucent.com ------- Issue 2155: OBV chunk boundaries (interop) Click here for this issue's archive. Nature: Uncategorized Issue Severity: Summary: Summary: Section 15.3.4.4. of ptc/98-10-11 includes the following statement wrt to ObV chunking: "The data may be split into multiple chunks at arbitrary points except within primitive CDR types, arrays of primitive types, strings, and wstrings." Unfortunately, this provides only dubious benfits, but introduces significant problems. Traditionally, CDR has been designed so that encoding engines are only responsible for marshalling CDR primitives, and remain independent of the actual constructed IDL type being marshalled. The above clause violates this rule: the CDR stream must know that the primitives being marshalled are part of an array, understand the size of the array, distinguish sequences from arrays, etc. In addition, this is impossible to implement using the Java portable streams.The streams are not informed of the IDL type being encoded, and cannot rely on arrays being marshalled using a single invocation of the write_x_array methods (or even using this method at all). Instead of this definition, I suggest that we allow chunks to end anywhere that a fragment could end. This simplifies the encoding rules (as there is no need for a marshalling engine to remember where a chunk can end vs. where a fragment can end), normalizes the definition, and becomes both possible and easy to implement using the Java portable streams. The above sentence would be rewritten to say: "The data may be split into multiple chunks in the same manner as fragments." Discussion: While this proposed change seems reasonable and like a good thing to do, we must encure that such a change will not break existing GIOP 1.2 unmarshalling implementations. Can existing GIOP 1.2 implementations cope with unmarshaling chunks which break in the middle of arrays, e.t.c? If this change would require a GIOP minor version change, then we still would have to live with the "marshaling problem" for GIOP 1.2 implementations. Is it that broken that we cannot live with the differences between fragment rules and chunk rules? Resolution: To be proposed after email discussion Revised Text: Actions taken: October 30, 1998: received issue March 5, 1999: moved from obv to interop ter Date: Tue, 07 Nov 2000 14:24:51 -0800 From: Everett Anderson X-Mailer: Mozilla 4.73 [en] (Windows NT 5.0; U) X-Accept-Language: en,pdf,ja MIME-Version: 1.0 To: "Rutt, T E (Tom)" CC: "'interop@Omg.org'" Subject: Re: Need Help on Issue 2155 Reso - OBV ChunckBoundary References: Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii X-UIDL: MU]!!n_Td9;6b!!D6Be9 Hi, I have some questions and comments. Question 1: What was the original intent of requiring that chunks NOT split arrays of primitives? I think it's because it is inefficient and unnecessary. Example of inefficiency: If you were chunking and writing an array of 8 byte primitives that will span two fragments AND you were allowed to end the chunk at the end of the fragment, then this would happen: Fragment i: [4 byte chunk length] [possibly other data here] [alignment for the 8 byte primitive array] [some 8 byte primitives] Fragment i+1: [GIOP 1.2 fragment header so you end on a 16 byte boundary] [4 byte chunk length] [** now need 4 bytes of extra alignment **] [rest of the 8 byte primitives] Another situation in which you can't just end the chunk at the end of a fragment is when writing a single primitive. Often the last thing in a GIOP 1.2 fragment is the alignment for a primitive. If you end your chunk at the end of the fragment, then in the next fragment, you must introduce a chunk length, possibly more alignment, and then the primitive itself. > Traditionally, CDR has been designed so that encoding engines are only > responsible for > marshalling CDR primitives, and remain independent of the actual constructed > IDL type > being marshalled. The above clause violates this rule: the CDR stream must > know that the > primitives being marshalled are part of an array, understand the size of the > array, > distinguish sequences from arrays, etc. Yes, the CDR stream must know that -- but that information is available. For instance, in Java, using the org.omg.CORBA.portable.InputStream and OutputStream, there are methods for writing the arrays of primitives. Why not use them? I think this is central to your question. Why is it necessary to avoid using these methods in Java? > In addition, this is impossible to > implement using > the Java portable streams.The streams are not informed of the IDL > type being > encoded, Question 2: What do you mean by portable streams? Do you mean the org.omg.CORBA.portable streams? We have implemented GIOP 1.2 with chunking and fragmentation in Java. It is possible, though the CDR stream does have to know that it is marshaling an array. It also has to know about fragment boundaries, as you know. Requiring ending of chunks at fragment boundaries makes implementation easier, but the encoding will be unnecessarily awkward and inefficient. I agree that an implementation may not be pretty given the amount of knowledge that has to be used by the CDR code -- chunking information including the size of the array and data type, fragment information, etc, but this information is available. It is possible to do it, and the result is a relatively compact wire format. - Everett Sender: jbiggar@corvette.floorboard.com Message-ID: <3A088A7C.8CAEF455@floorboard.com> Date: Tue, 07 Nov 2000 15:04:28 -0800 From: Jonathan Biggar X-Mailer: Mozilla 4.76 [en] (X11; U; SunOS 5.6 sun4u) X-Accept-Language: en MIME-Version: 1.0 To: Everett Anderson CC: "Rutt, T E (Tom)" , "'interop@Omg.org'" Subject: Re: Need Help on Issue 2155 Reso - OBV ChunckBoundary References: <3A088133.F86B6CE@sun.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii X-UIDL: Z'>!!0idd9dH_d9X?`!! Everett Anderson wrote: > Another situation in which you can't just end the chunk at the end of a > fragment is when writing a single primitive. Often the last thing in a > GIOP 1.2 fragment is the alignment for a primitive. Pardon me if I misunderstood you, but CDR alignment is done before a primitive, not after. So if I choose to end a chunk and fragment at the same place, why should there be any alignment bytes at the end of the fragment? -- Jon Biggar Floorboard Software jon@floorboard.com jon@biggar.org Date: Tue, 07 Nov 2000 15:29:50 -0800 From: Everett Anderson X-Mailer: Mozilla 4.73 [en] (Windows NT 5.0; U) X-Accept-Language: en,pdf,ja MIME-Version: 1.0 To: Jonathan Biggar CC: "Rutt, T E (Tom)" , "'interop@Omg.org'" Subject: Re: Need Help on Issue 2155 Reso - OBV ChunckBoundary References: <3A088133.F86B6CE@sun.com> <3A088A7C.8CAEF455@floorboard.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii X-UIDL: B]P!!a@%!!UM/!!(/*e9 > > Another situation in which you can't just end the chunk at the end of a > > fragment is when writing a single primitive. Often the last thing in a > > GIOP 1.2 fragment is the alignment for a primitive. > > Pardon me if I misunderstood you, but CDR alignment is done before a > primitive, not after. So if I choose to end a chunk and fragment at the Exactly. > same place, why should there be any alignment bytes at the end of the > fragment? You might end chunks and fragments in the same place by chance as things are, now. The problem would arise if we allowed chunks to end anywhere that a fragment could end, as was suggested. Let's say you have a fragment size of 32 bytes. The GIOP 1.2 fragment header takes 16 bytes. Due to other data, you are currently at byte 27, and now want to write an 8 byte primitive. First, let's consider the normal case -- no chunking. Fragment i [1.2 fragment header] [some data] [need all 5 bytes left for alignment of the primitive] Fragment i+1 [1.2 fragment header] [the 8 byte primitive] Now, if you end chunks at the end of the fragments in all cases, then a chunk that began in fragment i also has to end in fragment i. Thus, it would look like the following: Fragment i [1.2 fragment header] [some data, including chunk length, yet we're still at byte 27] [need all 5 byte left for alignment of the primitive] <-- chunk also ended here Fragment i+1 [1.2 fragment header] [need to open a new chunk length] [need to add 4 bytes of alignment for your 8 byte primitive. You already used 5 bytes previously for alignment, but the chunk length causes you to be off, again.] [the 8 byte primitive] So you've ultimately used 9 bytes of alignment and 4 bytes for chunk length when the current rules would yield 13 bytes less on the wire. Sender: jbiggar@corvette.floorboard.com Message-ID: <3A0894E8.5FD13281@floorboard.com> Date: Tue, 07 Nov 2000 15:48:56 -0800 From: Jonathan Biggar X-Mailer: Mozilla 4.76 [en] (X11; U; SunOS 5.6 sun4u) X-Accept-Language: en MIME-Version: 1.0 To: Everett Anderson CC: "Rutt, T E (Tom)" , "'interop@Omg.org'" Subject: Re: Need Help on Issue 2155 Reso - OBV ChunckBoundary References: <3A088133.F86B6CE@sun.com> <3A088A7C.8CAEF455@floorboard.com> <3A08906D.A0B5DC3C@sun.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii X-UIDL: CKL!!=>D!!`f$!!p9)!! Everett Anderson wrote: > > > > Another situation in which you can't just end the chunk at the end of a > > > fragment is when writing a single primitive. Often the last thing in a > > > GIOP 1.2 fragment is the alignment for a primitive. > > > > Pardon me if I misunderstood you, but CDR alignment is done before a > > primitive, not after. So if I choose to end a chunk and fragment at the > > Exactly. > > > same place, why should there be any alignment bytes at the end of the > > fragment? > > You might end chunks and fragments in the same place by chance as things > are, now. The problem would arise if we allowed chunks to end anywhere > that a fragment could end, as was suggested. Ok, I forgot that fragments (except the last one) have to have a multiple of 8 for the length. That can cause extra padding. -- Jon Biggar Floorboard Software jon@floorboard.com jon@biggar.org From: "Rutt, T E (Tom)" To: "'interop@omg.org'" Cc: "Rutt, T E (Tom)" Subject: wordsmithing Issue 2155 Proposal Date: Mon, 20 Nov 2000 13:25:48 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2650.21) Content-Type: text/plain X-UIDL: R<8e9eTW!!T1 X-Mailer: Mozilla 4.73 [en] (Windows NT 5.0; U) X-Accept-Language: en,pdf,ja MIME-Version: 1.0 To: "'interop@omg.org'" Subject: Re: wordsmithing Issue 2155 Proposal References: Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii X-UIDL: [Jld98='e9(L`d9#-ld9 Hi, I believe that this proposal will break CDR encoding with fragmentation. Please see my previous messages on this and the examples I gave so that there can be more discussion before the vote: http://cgi.omg.org/issues/issue2155.txt - Everett "Rutt, T E (Tom)" wrote: > > Please comment on the following proposal. If no comments are received, it > will go out > next week for vote: > > Issue 2155: OBV chunk boundaries (interop) > > Click here for this issue's archive. > Nature: Uncategorized Issue > Severity: > Summary: Summary: Section 15.3.4.4. of ptc/98-10-11 includes the following > statement wrt to ObV > chunking: "The data may be split into multiple chunks at arbitrary points > except within primitive CDR > types, arrays of primitive types, strings, and wstrings." Unfortunately, > this provides only dubious benfits, > but introduces significant problems. Traditionally, CDR has been designed so > that encoding engines are > only responsible for marshalling CDR primitives, and remain independent of > the actual constructed IDL > type being marshalled. The above clause violates this rule: the CDR stream > must know that the primitives > being marshalled are part of an array, understand the size of the array, > distinguish sequences from arrays, > etc. In addition, this is impossible to implement using the Java portable > streams.The streams are not > informed of the IDL type being encoded, > and cannot rely on arrays being marshalled using a single invocation of the > write_x_array methods (or > even using this method at all). > > Instead of this definition, I suggest that we allow chunks to end anywhere > that a fragment could end. This > simplifies the encoding rules (as there is no need for a marshalling engine > to remember where a chunk can > end vs. where > a fragment can end), normalizes the definition, and becomes both possible > and easy to implement using the > Java portable streams. > > The above sentence would be rewritten to say: > > "The data may be split into multiple chunks in the same manner as > fragments." > Discussion: > While this proposed change seems like a good thing to do, we must assume > that such a change will not > break exsiting GIOP 1.2 unmarshalling implementations if we do not rev GIOP > to 1.3. > > The proposed resolution assumes existing GIOP 1.2 implmentations can cope > with unmarshaling chunks > which break in the middle of arrays, e.t.c? > > Proposed Resolution: > To be proposed after email discussion > Revised Text: In Section 15.3.4.4. of ptc/98-10-11 Change: > " > The data may be split into multiple chunks at arbitrary points except within > primitive CDR types, arrays of > primitive types, strings, and wstrings. > " > to > " > The data may be split into multiple chunks in the same manner as fragments. > " > Actions taken: > October 30, 1998: received issue > March 5, 1999: moved from obv to interop > > Discussion: > ter