Issue 3151: Stream format problem for custom marshalled RMI types (java2idl-rtf) Source: (Mr. Simon C. Nash, ) Nature: Uncategorized Issue Severity: Summary: There is a problem in the stream format for custom marshalled RMI types. The Java serialization spec requires that when a class with a writeObject method calls defaultWriteObject and then writes optional data, the receiver must be able to skip the optional data if the readObject does not consume it or the class is not in the receiver's hierarchy. So if the sender's hierarchy consists of serializable classes Base and Derived, both of which call defaultWriteObject and then write optional data, there must be a way for the unmarshaller to first skip to the end of the Base data, then skip to the end of the Derived data. With the current stream format, this is not possible. Resolution: see below Revised Text: The following example shows a case that currently doesn't work: public class A implements java.io.Serializable { // any data } public class B extends A { // any data private void writeObject(java.io.ObjectOutputStream stream) { // any code } private void readObject(java.io.ObjectInputStream stream) { // any code } } public class C extends B { // any data } At present, an instance of C is marshalled as C-header A-data B-data C-data C-endtag where all the data is written and read by the ValueHandler and is chunked according to the valuetype chunking rules. Suppose some change to the writeObject() implementation in class B causes more B-data to be written, while everything else stays the same. When the old B code unmarshals a new B instance, it will not fully consume all the B-data and this will cause the unmarshalling of the C-data to be incorrect since the stream will be out of sync. The problem is caused by a deficiency in the stream encoding for custom serialized data. In the example above, the existing format does not identify the boundary between the B-data and the C-data, so it is impossible for the unmarshalling code in the ValueHandler to skip over any extraneous B-data before it starts to read the C-data. The proposed resolution is to enclose the B-data in a nested valuetype. The repid for the nested valuetype must be specified by the Java to IDL mapping so that it can be read by IDL custom unmarshalling code in other languages. The repid is "RMI:org.omg.custom.<class>:<hashcode>:<suid>" where <class> is the fully-qualified name of the class whose writeObject method is being invoked and <hashcode> and <suid> are the class's hashcode and SUID. The other language implementation has to provide a suitable custom valuetype implementation for the nested valuetype and may have to provide a value factory that maps the standard repid to this implementation. For Java unmarshalling, the ValueHandler will understand the special repid and process its contents as RMI-IIOP custom serialized data. Since valuetype headers and end tags are written by the ORB's OutputStream and not by the ValueHandler, it will be necessary to add new APIs to the OutputStream for the ValueHandler to call when marshalling custom valuetype data. These new APIs will write the valuetype header and end tag for the nested valuetype, keep track of the valuetype nesting level, and take care of chunking for the nested valuetype. Rather than adding yet another level of OutputStream subclassing, the new APIs are defined within an interface called org.omg.CORBA.portable.ValueOutputStream which will be implemented by the output stream passed to the ValueHandler. The new interface is defined as follows: public interface ValueOutputStream { void start_value(java.lang.String rep_id); void end_value(); } When the ValueHandler encounters a class with custom serialized data, it calls the start_value() method on the output stream that it was given, passing the special custom repid for the custom-serialized class. The output stream ends the currently open chunk, writes a valuetype header for the nested custom valuetype, and increments the valuetype nesting depth. It then returns to the ValueHandler, which calls writeObject() to write the custom serialized data. When writeObject() returns, the ValueHandler calls end_value() on the stream to end the current chunk, write the end tag, and decrement the valuetype nesting depth. When unmarshalling a class with custom serialized data, the ValueHandler needs to call the ORB's input stream to consume the valuetype header and end tag. A new interface org.omg.CORBA.portable.ValueInputStream that defines the necessary APIs will be implemented by the input stream. This interface is defined as follows: public interface ValueInputStream { void start_value(); void end_value(); } The ValueHandler calls the start_value() method on the input stream. The input stream reads the nested valuetype header and increments the valuetype nesting depth, then returns to the ValueHandler. The ValueHandler reads the custom serialized data, then calls end_value() which consumes the end tag (after skipping any data that precedes the end tag) and decrements the valuetype nesting depth. For languages other than Java, the nested valuetype must be handled explicitly by calling write_Value() and read_Value() within the implementations of the marshal() and unmarshal() methods. To assist in the implementation of these methods, a new IDL valuetype is generated by the Java to IDL mapping. In the example above, this would be: module org { module omg { module customRMI { custom valuetype B {}; #pragma ID B "RMI:org.omg.customRMI.B:1234567890ABCDEF:FEDCBA0987654321" }; }; }; This would be implemented in C++ (for example) with marshal() and unmarshal() methods that read and write the appropriate data. The new stream format would be identified by a format version of 2 (currently 1). By default, ORBs that support GIOP version 1.2 are assumed to accept format version 1 only, and ORBs that support GIOP 1.3 are assumed to accept format versions 1 and 2. These defaults can be overridden by a new tagged component for servers, and by a new service context for clients. The new tagged component is identified by the following component id: const ComponentId TAG_RMI_CUSTOM_MAX_STREAM_FORMAT = xx; // number to be assigned by OMG The TAG_RMI_CUSTOM_MAX_STREAM_FORMAT component has an associated value of type octet, encoded as a CDR encapsulation, designating the maximum stream format version for RMI/IDL custom value types that can be used in GIOP messages sent to this IOR. The TAG_RMI_CUSTOM_MAX_STREAM_FORMAT component can appear at most once in any IOR profile. For profiles supporting IIOP 1.2 or greater, it is optionally present. If this component is omitted, then the default maximum stream format version for RMI/IDL custom value types sent to this IOR is 1 for IIOP 1.2 and 2 for IIOP 1.3. The new service context is identified by the following service id: const ServiceId RMICustomMaxStreamFormat = yy; // number to be assigned by OMG RMICustomMaxStreamFormat identifies a CDR encapsulation of a single octet that specifies the highest RMI/IDL custom stream format version that can be used for RMI/IDL custom valuetypes marshaled within a GIOP reply associated with the GIOP request that carries this service context.Revised Text: The following changes are proposed in the Java to IDL mapping specification, document number ptc/2001-06-15: In section 1.3.5.6, add the following at the end of the second paragraph: "An additional IDL custom valuetype in the module ::org::omg::customRMI is also generated to assist with marshaling and unmarshaling instances of the class. See section 1.3.5.8 for details." Add a new section 1.3.5.8 "Secondary custom valuetype", renumbering the existing section 1.3.5.8 and subsequent sections: "In addition to the primary mapping described above, an RMI/IDL value type containing a writeObject method is mapped to a secondary IDL custom valuetype. The module name for this valuetype is formed by taking the ::org::omg::customRMI prefix and then adding the primary mapped type’s module name. The name of the secondary valuetype is the same as the name of the primary IDL custom value type to which the RMI/IDL value type was mapped. The secondary valuetype has no inheritance, data members, methods, or initializers. It has a #pragma ID specifying a repository ID formed by taking the repository ID of the primary custom valuetype and prefixing the Java package name with "org.omg.customRMI.". The secondary custom valuetype represents the enclosure of writeObject data that is wriitten to the serialization stream when the primary custom valuetype or any of its subclasses is serialized using format version 2, as described in item 1d of section 1.4.10. For IDL custom marshaling and unmarshaling of the primary mapped IDL valuetype, the marshal() and unmarshal() methods can call write_Value() and read_Value() to write and read the nested valuetype enclosure. This will cause the marshal() and unmarshal() methods of the secondary mapped IDL valuetype to be called to write and read the custom serialized data." In existing section 1.3.5.9, add the following to the generated IDL: module org { module omg { module customRMI { module alpha { module bravo { custom valuetype Kangaroo {}; #pragma ID Kangaroo "RMI:org.omg.customRMI.alpha.bravo.Kangaroo:87654321ABCDEF01:9876543210FEDCBA" }; }; }; }; }; In section 1.4.10, replace numbered paragraphs 1 and 2 by the following: "a. octet - Format version. 1 or 2. For serializable objects with a writeObject method: b. boolean - True if defaultWriteObject was called, false otherwise. c. (optional) Data written by defaultWriteObject. The ordering of the fields is the same as the order in which they appear in the mapped IDL valuetype, and these fields are encoded exactly as they would be if the class did not have a writeObject method. d. (optional) Additional data written by writeObject, encoded as specified below. For format version 1, the data is written "as is". For format version 2, the data is enclosed within a CDR custom valuetype with no codebase and repid "RMI:org.omg.custom.<class>" where <class> is the fully-qualified name of the class whose writeObject method is being invoked. For externalizable objects: b. (optional) Data written by writeExternal, encoded as specified below." In the same section, at the end of subsection 1, add the following: "The default custom stream format is 1 for GIOP 1.2 and 2 for GIOP 1.3. For RMI/IDL custom value types marshaled within GIOP requests, a format version not greater than the default for the GIOP message level must be sent, except where the TAG_RMI_CUSTOM_MAX_STREAM_FORMAT tagged component (see section 1.4.11) is part of the IOR profile. For RMI/IDL custom value types marshaled within GIOP replies (including the UnknownExceptionInfo service context), a format version not greater than the default for the GIOP message level must be sent, except where the RMICustomMaxStreamFormat service context (see section 1.4.12) was sent on the associated GIOP request." Add a new section 1.4.11 "TAG_RMI_CUSTOM_MAX_STREAM_FORMAT Component", renumbering the existing section 1.4.11 and subsequent sections: "Although the IIOP level of an IOR specifies a default maximum stream format version for RMI/IDL custom value types marshaled as part of GIOP requests to this IOR, there are cases when it may be necessary to override this default. The TAG_RMI_CUSTOM_MAX_STREAM_FORMAT component has an associated value of type octet, encoded as a CDR encapsulation, designating the maximum stream format version for RMI/IDL custom value types that can be used in GIOP messages sent to this IOR. The TAG_RMI_CUSTOM_MAX_STREAM_FORMAT component can appear at most once in any IOR profile. For profiles supporting IIOP 1.2 or greater, it is optionally present. If this component is omitted, then the default maximum stream format version for RMI/IDL custom value types sent to this IOR is 1 for IIOP 1.2 and 2 for IIOP 1.3." Add a new section 1.4.12 "RMICustomMaxStreamFormat Service Context", renumbering the existing section 1.4.11 and subsequent sections: "Although the GIOP level of a request specifies a default maximum stream format version for RMI/IDL custom value types marshaled as part of the associated reply, there are cases when it may be necessary to override this default. RMICustomMaxStreamFormat identifies a CDR encapsulation of a single octet that specifies the highest RMI/IDL custom stream format version that can be used for RMI/IDL custom valuetypes marshaled within a GIOP reply associated with the GIOP request that carries this service context. If this service context is omitted from a GIOP request, then the default maximum stream format version for RMI/IDL custom value types marshaled within a GIOP reply associated with this request is 1 for GIOP 1.2 and 2 for GIOP 1.3." Change the title of existing section 1.5.1.3 to "ValueHandler and ValueHandlerMultiFormat". Also change the first paragraph to: "The interfaces javax.rmi.CORBA.ValueHandler and javax.rmi.CORBA.ValueHandlerMultiFormat define methods that allow serialization of Java objects to and from GIOP streams." Also add the following interface: //Java public interface ValueHandlerMultiFormat extends ValueHandler { byte getMaximumStreamFormatVersion(); void writeValue(org.omg.CORBA.portable.OutputStream out, java.io.Serializable value, byte streamFormatVersion); } and add the following paragraphs to this section before the "Execution Model for Serialization" subsection: "The ValueHandlerMultiFormat interface introduces a method getMaximumStreamFormatVersion that returns the maximum stream format version for RMI/IDL custom value types that is supported by this ValueHandler object. The ValueHandler object must support the returned stream format version and all lower versions. The format versions currently defined are 1 and 2. See section 1.4.10 "Custom Marshaling Format" for more details. The ValueHandlerMultiFormat interface introduces an overloaded writeValue method that allows the ORB to pass the required stream format version for RMI/IDL custom value types. If the ORB calls this method, it must pass a stream format version between 1 and the value returned by the getMaximumStreamFormatVersion method inclusive, or else a BAD_PARAM exception with standard minor code aa (note to editor: number to be assigned by OMG) must be thrown. If the ORB calls the ValueHandler.writeValue method, stream format version 1 is implied." Add new sections 1.5.1.3 "ValueOutputStream" and 1.5.1.4 "ValueInputStream", renumbering the existing section 1.5.1.3 and subsequent sections: "The interface org.omg.CORBA.portable.ValueOutputStream defines methods that allow serialization of custom-marshaled RMI/IDL objects to GIOP streams. // Java package org.omg.CORBA.portable; public interface ValueOutputStream { void start_value(java.lang.String rep_id); void end_value(); } The interface org.omg.CORBA.portable.ValueInputStream defines methods that allow deserialization of custom-marshaled RMI/IDL objects from GIOP streams. // Java package org.omg.CORBA.portable; public interface ValueInputStream { void start_value(); void end_value(); } " In existing section 1.5.1.3, add a new paragraph at the end of the "Execution Model for Serialization" section as follows: "The ORB output stream passed to the ValueHandlerMultiFormat.writeValue method must implement the ValueOutputStream interface, and the ORB input stream passed to the ValueHandler.readValue method must implement the ValueInputStream interface. For output streams, the start_value() method ends any currently open chunk, writes a valuetype header for a nested custom valuetype (with a null codebase and the specified repository ID), and increments the valuetype nesting depth. The end_value() method ends any currently open chunk, writes the end tag for the nested custom valuetype, and decrements the valuetype nesting depth. For input streams, the start_value() method reads a valuetype header for a nested custom valuetype and increments the valuetype nesting depth. The end_value() method reads the end tag for the nested custom valuetype (after skipping any data that precedes the end tag) and decrements the valuetype nesting depth." In existing section 1.5.1.3, add new paragraphs at the end of the "Value Marshaling" section as follows: "Before calling the writeValue method of the ValueHandler object, the ORB must determine the stream format version to be used. This is the maximum format version that is supported by both the local ValueHandler and the remote connection endpoint. The maximum local format version is the value returned by the getMaximumStreamFormatVersion method of the ValueHandler object, or 1 if the ValueHandler object doesn't support the ValueHandlerMultiFormat interface. The maximum remote format version is 1 for GIOP 1.2 messages and 2 for GIOP 1.3 messages, except where these default values are overridden by either the TAG_RMI_CUSTOM_MAX_STREAM_FORMAT tagged component (see section 1.4.11) or the RMICustomMaxStreamFormat service context (see section 1.4.12). For GIOP 1.2 messages, recognition of these overrides is optional. If the stream format version computed in this way is 2 or greater, the ORB must call the ValueHandlerMultiFormat.writeValue method, passing this value. If the stream format version computed in this way is 1, the ORB may call either the ValueHandlerMultiFormat.writeValue method (with stream format 1) or the ValueHandler.writeValue method. If the ORB's call to the ValueHandler object's writeValue method specified RMI/IDL custom value type stream format version 2, then the ValueHandler object must call the ValueOutputStream.start_value() and ValueOutputStream.end_value() methods of the ORB stream before and after writing the data specified by item 1d of section 1.4.10. The rep_id string passed to the start_value() method must be "RMI:org.omg.custom.<class>:<hashcode>:<suid>" where <class> is the fully-qualified name of the class whose writeObject method is being invoked and <hashcode> and <suid> are the class's hashcode and SUID. For format version 2, if the ORB stream passed to the ValueHandler doesn't support the ValueOutputStream interface, then a BAD_PARAM exception with standard minor code bb (note to editor: number to be assigned by OMG) must be thrown." In existing section 1.5.1.3, add a new paragraph at the end of the "Value Unmarshaling" section as follows: "If the RMI/IDL custom data unmarshaled from the input stream was encoded using stream format 2, then the ValueHandler object must call the ValueInputStream.start_value() and ValueInputStream.end_value() methods of the ORB stream before and after reading the data specified by item 1d of section 1.4.10. For format version 2, if the ORB stream passed to the ValueHandler doesn't support the ValueInputStream interface, then a BAD_PARAM exception with standard minor code cc (note to editor: number to be assigned by OMG) must be thrown. If the format version unmarshaled by the ValueHandler is greater than the maximum version that it supports, then a MARSHAL exception with standard minor code dd (note to editor: number to be assigned by OMG) must be thrown." The following changes are proposed in the CORBA specification, document number formal/2001-09-34: In section 4.11.4, add the following exception standard minor codes: BAD_PARAM aa (note to editor: number to be assigned by OMG) Unsupported RMI/IDL custom value type stream format. BAD_PARAM bb (note to editor: number to be assigned by OMG) ORB output stream does not support ValueOutputStream interface. BAD_PARAM cc (note to editor: number to be assigned by OMG) ORB input stream does not support ValueInputStream interface. MARSHAL dd (note to editor: number to be assigned by OMG) Unsupported RMI/IDL custom value type stream format. In section 13.6.6, add the following ComponentId constant definition to the first list: const ComponentId TAG_RMI_CUSTOM_MAX_STREAM_FORMAT = xx; // number to be assigned by OMG In section 13.6.6.3, add the following bullet: TAG_RMI_CUSTOM_MAX_STREAM_FORMAT - See the Java to IDL mapping specification (formal/xx-xx-xx), section 1.4.11 "TAG_RMI_CUSTOM_MAX_STREAM_FORMAT Component". In section 13.7.1, add the following ServiceId constant definition: const ServiceId RMICustomMaxStreamFormat = yy; // number to be assigned by OMG and the following bullet: RMICustomStreamFormat identifies a CDR encapsulation of a single octet as described in the Java to IDL mapping specification (formal/xx-xx-xx), section 1.4.12 "RMICustomMaxStreamFormat Service Context". In section 13.9, add a row to the table for feature "RMICustomStreamFormat service context" showing that it is optional for version 1.2 and mandatory for GIOP 1.3 (when the 1.3 column is added). In section A.2, add a row to the table for ServiceId tag value RMICustomMaxStreamFormat indicating that it is described in the Java to IDL mapping specification (formal/xx-xx-xx), section 1.4.12 "RMICustomMaxStreamFormat Service Context". In section A.3, add a row to the table for ComponentId tag value TAG_RMI_CUSTOM_MAX_STREAM_FORMAT indicating that it is described in the Java to IDL mapping specification (formal/xx-xx-xx), section 1.4.11 "TAG_RMI_CUSTOM_MAX_STREAM_FORMAT Component". Actions taken: December 20, 1999: received issue May 13, 2002: closed issue Discussion: End of Annotations:===== Date: Sun, 19 Dec 1999 21:31:20 +0000 From: Simon Nash Organization: IBM X-Mailer: Mozilla 4.5 [en] (Win98; I) X-Accept-Language: en MIME-Version: 1.0 To: issues@omg.org CC: java2idl-rtf@omg.org Subject: Stream format problem for custom marshalled RMI types Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii X-UIDL: ~]&e9#,G!!Y_5e9W2k!! There is a problem in the stream format for custom marshalled RMI types. The Java serialization spec requires that when a class with a writeObject method calls defaultWriteObject and then writes optional data, the receiver must be able to skip the optional data if the readObject does not consume it or the class is not in the receiver's hierarchy. So if the sender's hierarchy consists of serializable classes Base and Derived, both of which call defaultWriteObject and then write optional data, there must be a way for the unmarshaller to first skip to the end of the Base data, then skip to the end of the Derived data. With the current stream format, this is not possible. The JRMP (serialization) format allows this, since it has a TC_ENDBLOCKDATA marker for every class. The equivalent marker in RMI-IIOP is the valuetype end tag, and this occurs only at the end of the object, not at the end of each class's data. We therefore need another marker that indicates the end of each class's data. Because of the separation of concerns between the ValueHandler and the ORB stream, this marker must be writable by the ValueHandler since only it knows the class structure of the object being marshalled. Another aspect of the same problem is that the current stream format does not prevent a custom unmarshaller's readObject method reading past the end of the data written by the writeObject method of the same class. One way to fix this (suggested by Bob) is as follows: 1. Introduce a new stream format ID of 2. 2. For format 2, all the optional data written by a class is enclosed in a chunked valuetype with no repid and no codebase. 3. When the unmarshaller is reading a class's optional data, it can skip to the end of it by doing the following: a. read non-valuetype data a byte at a time until the stream throws an exception saying that a nested valuetype was encountered or that there is no more data available to consume in the current valuetype. b. if a nested valuetype was encountered, read nested valuetypes until the stream throws an exception saying that non-valuetype data was encountered or that there is no more data available to consume in the current valuetype. c. if non-valuetype data was encountered, go back to step a. Simon -- Simon C Nash, Technology Architect, IBM Java Technology Centre Tel. +44-1962-815156 Fax +44-1962-818999 Hursley, England Internet: nash@hursley.ibm.com Lotus Notes: Simon Nash@ibmgb Date: Mon, 27 Nov 2000 11:20:26 +0000 From: Simon Nash Organization: IBM X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; I) X-Accept-Language: en MIME-Version: 1.0 To: java2idl-rtf@omg.org Subject: Issue 3151 proposed resolution Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii X-UIDL: 7pM!!QAa!!nC Reply-To: Bob Scheifler - SMI Software Development Subject: Re: Issue 3151 proposed resolution To: nash@hursley.ibm.com Cc: java2idl-rtf@omg.org MIME-Version: 1.0 Content-MD5: smXAvoaDquSGmBlICceCnw== X-Mailer: dtmail 1.2.1 CDE Version 1.2.1 SunOS 5.6 sun4u sparc Content-Type: TEXT/plain; charset=us-ascii X-UIDL: 'Q%e99&:!!$Ba!!)hTd9 > 1. Introduce a new stream format ID of 2. Some things don't seem to be spelled out for either option. Are conforming implementations required to understand both the old and the new format? Are different formats permitted to coexist within the same call? How does a client reliably communicate with a server when it doesn't know which formats are supported by the server? Is there any sort of format negotiation, query, or retry? - Bob Date: Thu, 30 Nov 2000 19:28:14 +0000 From: Simon Nash Organization: IBM X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; I) X-Accept-Language: en MIME-Version: 1.0 To: Bob Scheifler - SMI Software Development CC: java2idl-rtf@omg.org Subject: Re: Issue 3151 proposed resolution References: <200011282339.SAA27472@eastmail2.East.Sun.COM> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii X-UIDL: :GYd9p4[!!`91e95UPd9 Bob, As always, you ask good questions. These issues apply to both options A and B, so they should not influence the choice between them. Here are my sugggested answers. Compliant receivers must accept format 1 and may accept format 2. Compliant senders must be able to send format 1 and may optionally send format 2. Sending a mixture of formats within the same call is allowed. There is no standard mechanism for format negotiation. The means by which a sender decides which format to send is implementation-dependent. However, compliant senders must provide a way to force them to send format 1 when talking to a receiver that can only understand this format. Retry is not recommended since it only works in the client to server direction. A client could send the "wrong" format to a server, get an exception back, and retry using the other format. However when a server sends a reply to a client, if this is in the "wrong" format then there is no way for the client to inform the server of this so that the server can retry. These rules allow existing implementations to continue to be compliant, and ensure that updated implementations that support format 2 will always be able to interoperate with existing implementations that only support format 1. Any comments? Simon Bob Scheifler - SMI Software Development wrote: > > > 1. Introduce a new stream format ID of 2. > > Some things don't seem to be spelled out for either option. Are > conforming implementations required to understand both the old > and the new format? Are different formats permitted to coexist > within the same call? How does a client reliably communicate with > a server when it doesn't know which formats are supported by the > server? Is there any sort of format negotiation, query, or retry? > > - Bob -- Simon C Nash, Technology Architect, IBM Java Technology Centre Tel. +44-1962-815156 Fax +44-1962-818999 Hursley, England Internet: nash@hursley.ibm.com Lotus Notes: Simon Nash@ibmgb Date: Thu, 30 Nov 2000 18:43:04 -0500 (EST) From: Bob Scheifler - SMI Software Development Reply-To: Bob Scheifler - SMI Software Development Subject: Re: Issue 3151 proposed resolution To: nash@hursley.ibm.com Cc: java2idl-rtf@omg.org MIME-Version: 1.0 Content-MD5: Hfu42TJsDPqYNurgkRgcTg== X-Mailer: dtmail 1.2.1 CDE Version 1.2.1 SunOS 5.6 sun4u sparc Content-Type: TEXT/plain; charset=us-ascii X-UIDL: 8Y@!!gU5e9OoGe9-F+!! > Option B: > The marker object can be any serializable object whose class is part > of > the core Java platform. Does it need to be a class that has no writeObject method, to avoid infinite recursion? - Bob Date: Thu, 30 Nov 2000 18:13:48 -0800 From: "Vijaykumar Natarajan" X-Mailer: Mozilla 4.51 [en]C-CCK-MCD (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: Simon Nash CC: Bob Scheifler - SMI Software Development , java2idl-rtf@omg.org Subject: Re: Issue 3151 proposed resolution References: <200011282339.SAA27472@eastmail2.East.Sun.COM> <3A26AA4E.19D38F2C@hursley.ibm.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii X-UIDL: "]Od90NPe9)d3!!Gm/!! Hi Simon, I am afraid, I will be unable to spend the time to look at this right away as it requires some attention and thought. Could we please look at this first thing next RTF as opposed to trying to rush it in this time. Thanks, Vijay Simon Nash wrote: > Bob, > As always, you ask good questions. These issues apply to both > options > A and B, so they should not influence the choice between them. Here > are > my sugggested answers. > > Compliant receivers must accept format 1 and may accept format 2. > Compliant senders must be able to send format 1 and may optionally > send > format 2. Sending a mixture of formats within the same call is > allowed. > > There is no standard mechanism for format negotiation. The means by > which > a sender decides which format to send is implementation-dependent. > However, > compliant senders must provide a way to force them to send format 1 > when > talking to a receiver that can only understand this format. > Retry is not recommended since it only works in the client to server > direction. A client could send the "wrong" format to a server, get > an > exception back, and retry using the other format. However when a > server > sends a reply to a client, if this is in the "wrong" format then > there is > no way for the client to inform the server of this so that the > server > can retry. > > These rules allow existing implementations to continue to be > compliant, > and ensure that updated implementations that support format 2 will > always be able to interoperate with existing implementations that > only > support format 1. > > Any comments? > > Simon > > Bob Scheifler - SMI Software Development wrote: > > > > > 1. Introduce a new stream format ID of 2. > > > > Some things don't seem to be spelled out for either option. Are > > conforming implementations required to understand both the old > > and the new format? Are different formats permitted to coexist > > within the same call? How does a client reliably communicate with > > a server when it doesn't know which formats are supported by the > > server? Is there any sort of format negotiation, query, or retry? > > > > - Bob > > -- > Simon C Nash, Technology Architect, IBM Java Technology Centre > Tel. +44-1962-815156 Fax +44-1962-818999 Hursley, England > Internet: nash@hursley.ibm.com Lotus Notes: Simon Nash@ibmgb Date: Fri, 01 Dec 2000 12:17:53 +0000 From: Simon Nash Organization: IBM X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; I) X-Accept-Language: en MIME-Version: 1.0 To: Vijaykumar Natarajan CC: Bob Scheifler - SMI Software Development , java2idl-rtf@omg.org Subject: Re: Issue 3151 proposed resolution References: <200011282339.SAA27472@eastmail2.East.Sun.COM> <3A26AA4E.19D38F2C@hursley.ibm.com> <3A27095C.7FCC5EB3@inprise.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii X-UIDL: XY>e9gK > Hi Simon, > > I am afraid, I will be unable to spend the time to look at this right away as it > requires some attention and thought. Could we please look at this first thing > next RTF as opposed to trying to rush it in this time. > > Thanks, > Vijay > > Simon Nash wrote: > > > Bob, > > As always, you ask good questions. These issues apply to both options > > A and B, so they should not influence the choice between them. Here are > > my sugggested answers. > > > > Compliant receivers must accept format 1 and may accept format 2. > > Compliant senders must be able to send format 1 and may optionally send > > format 2. Sending a mixture of formats within the same call is allowed. > > > > There is no standard mechanism for format negotiation. The means by which > > a sender decides which format to send is implementation-dependent. However, > > compliant senders must provide a way to force them to send format 1 when > > talking to a receiver that can only understand this format. > > > Retry is not recommended since it only works in the client to server > > direction. A client could send the "wrong" format to a server, get an > > exception back, and retry using the other format. However when a server > > sends a reply to a client, if this is in the "wrong" format then there is > > no way for the client to inform the server of this so that the server > > can retry. > > > > These rules allow existing implementations to continue to be compliant, > > and ensure that updated implementations that support format 2 will > > always be able to interoperate with existing implementations that only > > support format 1. > > > > Any comments? > > > > Simon > > > > Bob Scheifler - SMI Software Development wrote: > > > > > > > 1. Introduce a new stream format ID of 2. > > > > > > Some things don't seem to be spelled out for either option. Are > > > conforming implementations required to understand both the old > > > and the new format? Are different formats permitted to coexist > > > within the same call? How does a client reliably communicate with > > > a server when it doesn't know which formats are supported by the > > > server? Is there any sort of format negotiation, query, or retry? > > > > > > - Bob > > > > -- > > Simon C Nash, Technology Architect, IBM Java Technology Centre > > Tel. +44-1962-815156 Fax +44-1962-818999 Hursley, England > > Internet: nash@hursley.ibm.com Lotus Notes: Simon Nash@ibmgb -- Simon C Nash, Technology Architect, IBM Java Technology Centre Tel. +44-1962-815156 Fax +44-1962-818999 Hursley, England Internet: nash@hursley.ibm.com Lotus Notes: Simon Nash@ibmgb Date: Fri, 01 Dec 2000 12:19:03 +0000 From: Simon Nash Organization: IBM X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; I) X-Accept-Language: en MIME-Version: 1.0 To: Bob Scheifler - SMI Software Development CC: java2idl-rtf@omg.org Subject: Re: Issue 3151 proposed resolution References: <200011302339.SAA22091@eastmail2.East.Sun.COM> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii X-UIDL: 0<3e9_:c!!p;8e9c]L!! Bob, Yes, this would be necessary. Simon Bob Scheifler - SMI Software Development wrote: > > > Option B: > > The marker object can be any serializable object whose class is part of > > the core Java platform. > > Does it need to be a class that has no writeObject method, to avoid > infinite recursion? > > - Bob -- Simon C Nash, Technology Architect, IBM Java Technology Centre Tel. +44-1962-815156 Fax +44-1962-818999 Hursley, England Internet: nash@hursley.ibm.com Lotus Notes: Simon Nash@ibmgb Date: Fri, 1 Dec 2000 13:58:02 -0500 (EST) From: Bob Scheifler - SMI Software Development Reply-To: Bob Scheifler - SMI Software Development Subject: Re: Issue 3151 proposed resolution To: nash@hursley.ibm.com Cc: java2idl-rtf@omg.org MIME-Version: 1.0 Content-MD5: uD1mTSe+uKfKKdPpLUNNAQ== X-Mailer: dtmail 1.2.1 CDE Version 1.2.1 SunOS 5.6 sun4u sparc Content-Type: TEXT/plain; charset=us-ascii X-UIDL: m^@!!\cIe9F$:e9\W0e9 Simon, your writeup attempts to characterize the relative implementation complexity of the two options, which is good. However, it seems to me that the more important comparision for choosing between them is the relative performance hit, both in the extra bytes on the wire and the extra time to generate them. Is it possible to characterize these? - Bob Date: Sun, 14 Oct 2001 21:14:53 +0100 From: Simon Nash Organization: IBM X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; I) X-Accept-Language: en MIME-Version: 1.0 To: Mary Leland , Vijay Natarajan , Stefan Bauer , Jeff Mischkinsky , Yoshitaka Honishi , Bob Scheifler CC: java2idl-rtf@omg.org Subject: Issue 3151 proposed resolution Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=iso-8859-1 X-UIDL: /Yld9f]S!!Oo(!!"j^!! Since issue 3151 has been deferred by the last two RTFs, I am very much hoping that we will be able to resolve it this time. Here's an example of the case that doesn't work: public class A implements java.io.Serializable { // any data } public class B extends A { // any data private void writeObject(java.io.ObjectOutputStream stream) { // any code } private void readObject(java.io.ObjectInputStream stream) { // any code } } public class C extends B { // any data } At present, an instance of C is marshalled as C-header A-data B-data C-data C-endtag where all the data is written and read by the ValueHandler and is chunked according to the valuetype chunking rules. Suppose some change to the writeObject() implementation in class B cause secondary valuetype is the same as the name of the primary IDL custom value type to which the RMI/IDL value type was mapped. The secondary valuetype has no inheritance, data members, methods, or initializers. It has a #pragma ID specifying a repository ID formed by taking the repository ID of the primary custom valuetype and prefixing the Java package name with "org.omg.customRMI.". The secondary custom valuetype represents the enclosure of writeObject data that is wriitten to the serialization stream when the primary custom valuetype or any of its subclasses is serialized using format version 2, as described in item 1d of section 1.4.10. For IDL custom marshaling and unmarshaling of the primary mapped IDL valuetype, the marshal() and unmarshal() methods can call write_Value() and read_Value() to write and read the nested valuetype enclosure. This will cause the marshal() and unmarshal() methods of the secondary mapped IDL valuetype to be called to write and read the custom serialized data. 3. In existing section 1.3.5.9, add the following to the generated IDL: module org { module omg { module customRMI { module alpha { module bravo { custom valuetype Kangaroo {}; #pragma ID Kangaroo "RMI:org.omg.customRMI.alpha.bravo.Kangaroo:87654321ABCDEF01:9876543210FEDCBA" }; }; }; }; }; 4. In section 1.4.10, change paragraph 1a to read: a. octet - Format version. 1 or 2. 5. In the same section, change paragraph 1d to read: d. (optional) Additional data written by writeObject, encoded as specified below. For format version 1, the data is written "as is". For format version 2, the data is enclosed within a CDR custom valuetype with no codebase and repid "RMI:org.omg.custom." where is the fully-qualified name of the class whose writeObject method is being invoked. 6. In the same section, at the end of subsection 1, add the following: For interoperabilty using GIOP 1.2, format version 1 is required. 7. Add a new section 1.5.1.3 "ValueStream", renumbering the existing section 1.5.1.3 and subsequent sections: The interface org.omg.CORBA.portable.ValueStream defines methods that allow serialization of custom-marshaled RMI/IDL objects to and from GIOP streams. // Java package org.omg.CORBA.portable; public interface ValueStream { void start_value(java.lang.String rep_id); void end_value(); } 8. In existing section 1.5.1.3, add a new paragraph at the end of the "Execution Model for Serialization" section as follows: If the ORB supports custom serialization format 2 as described in item 1a of section 1.4.10, then the ORB streams passed to the readValue and writeValue methods of the ValueHandler object must implement the ValueStream interface. For output streams, the start_value() method ends any currently open chunk, writes a valuetype header for a nested custom valuetype (with a null codebase and the specified repository ID), and increments the valuetype nesting depth. The end_value() method ends any currently open chunk, writes the end tag for the nested custom valuetype, and decrements the valuetype nesting depth. For input streams, the start_value() method reads a valuetype header for a nested custom valuetype and increments the valuetype nesting depth. The end_value() method reads the end tag for the nested custom valuetype and decrements the valuetype nesting depth. 9. In existing section 1.5.1.3, add a new paragraph at the end of the "Value Marshaling" section as follows: If the ValueHandler supports custom serialization format 2 as described in item 1a of section 1.4.10, then the ValueHandler object must call the ValueStream.start_value() and ValueStream.end_value() methods of the ORB stream before and after writing the data specified by item 1d of section 1.4.10. The rep_id string passed to the start_value() method must be "RMI:org.omg.custom.::" where is the fully-qualified name of the class whose writeObject method is being invoked and and are the class's hashcode and SUID. 10. In existing section 1.5.1.3, add a new paragraph at the end of the "Value Unmarshaling" section as follows: If the ValueHandler supports custom serialization format 2 as described in item 1a of section 1.4.10, then the ValueHandler object must call the ValueStream.start_value() and ValueStream.end_value() methods of the ORB stream before and after reading the data specified by item 1d of section 1.4.10. The rep_id string passed to the start_value() method must be null. I apologize for the length and complexity of this proposal. As Einstein is famously quoted as saying: "Things should be as simple as possible, but no simpler." I believe this proposal meets that test, but I'm happy to discuss any alternatives that RTF members would like to put forward. The best way forward might be to schedule a conference call to go over the issue and my proposal in detail and discuss any alternative ideas for resolving this issue. Your feedback would be much appreciated. Simon -- Simon C Nash, Chief Technical Officer, IBM Java Technology Tel. +44-1962-815156 Fax +44-1962-818999 Hursley, England Internet: nash@hursley.ibm.com Lotus Notes: Simon Nash@ibmgb s more B-data to be written, while everything else stays the same. When the old B code unmarshals a new B instance, it will not fully consume all the B-data and this will cause the unmarshalling of the C-data to be incorrect since the stream will be out of sync. The problem is caused by a deficiency in the stream encoding for custom serialized data. In the example above, the existing format does not identify the boundary between the B-data and the C-data, so it is impossible for the unmarshalling code in the ValueHandler to skip over any extraneous B-data before it starts to read the C-data. Having carefully considered a number of possible solutions to this problem, I think Bob Scheifler's proposal of enclosing the B-data in a nested valuetype is the best approach. The repid for the nested valuetype must be specified by the Java to IDL mapping so that it can be read by IDL custom unmarshalling code in other languages. I propose that this repid be "RMI:org.omg.custom.::" where is the fully-qualified name of the class whose writeObject method is being invoked and and are the class's hashcode and SUID. The other language will have to provide a suitable custom valuetype implementation for the nested valuetype and may have to provide a value factory that maps the standard repid to this implementation. For Java unmarshalling, the ValueHandler will understand the special repid and process its contents as RMI-IIOP custom serialized data. Since valuetype headers and end tags are written by the ORB's OutputStream and not by the ValueHandler, it will be necessary to add new APIs to the OutputStream for the ValueHandler to call when marshalling custom valuetype data. These new APIs will write the valuetype header and end tag for the nested valuetype, keep track of the valuetype nesting level, and take care of chunking for the nested valuetype. Rather than adding yet another level of OutputStream subclassing, I propose adding the new APIs to an interface called org.omg.CORBA.portable.ValueStream which will be implemented by the output stream passed to the ValueHandler. The new interface is defined as follows: public interface ValueStream { void start_value(java.lang.String rep_id); void end_value(); } When the ValueHandler encounters a class with custom serialized data, it calls the start_value() method on the output stream that it was given, passing the special custom repid for the custom-serialized class. The output stream ends the currently open chunk, writes a valuetype header for the nested custom valuetype, and increments the valuetype nesting depth. It then returns to the ValueHandler, which calls writeObject() to write the custom serialized data. When writeObject() returns, the ValueHandler calls end_value() on the stream to end the current chunk, write the end tag, and decrement the valuetype nesting depth. When unmarshalling a class with custom serialized data, the ValueHandler needs to call the ORB's input stream to consume the valuetype header and end tag. The ValueStream interface can also be used for this purpose. The ValueHandler calls the start_value() method on the input stream, passing null as the repid. The input stream reads the nested valuetype header and increments the valuetype nesting depth, then returns to the ValueHandler. The ValueHandler reads the custom serialized data, then calls end_value() which consumes the end tag and decrements the valuetype nesting depth. For languages other than Java, the nested valuetype must be handled explicitly by calling write_Value() and read_Value() within the implementations of the marshal() and unmarshal() methods. To assist in the implementation of these methods, a new IDL valuetype is generated by the Java to IDL mapping. In the example above, this would be: module org { module omg { module customRMI { custom valuetype B {}; #pragma ID B "RMI:org.omg.customRMI.B:1234567890ABCDEF:FEDCBA0987654321" }; }; }; This would be implemented in C++ (for example) with marshal() and unmarshal() methods that read and write the appropriate data. The new stream format would be identified by a format version of 2 (currently 1). Because GIOP 1.2 uses format version 1, support for format version 2 cannot be made mandatory until GIOP 1.3. However, format version 2 can still be defined as an option for ORBs to support if they know they are talking to a partner ORB that understands this format. This could be via some proprietary means such as a vendor-specific service context or a configuration property. Here's a straw-man proposal for the changes needed to the Java to IDL mapping spec, document number ptc/2001-06-15: 1. In section 1.3.5.6, add the following at the end of the second paragraph: An additional IDL custom valuetype in the module ::org::omg::customRMI is also generated to assist with marshaling and unmarshaling instances of the class. See section 1.3.5.8 for details. 2. Add a new section 1.3.5.8 "Secondary custom valuetype", renumbering the existing section 1.3.5.8 and subsequent sections: In addition to the primary mapping described above, an RMI/IDL value type containing a writeObject method is mapped to a secondary IDL custom valuetype. The module name for this valuetype is formed by taking the ::org::omg::customRMI prefix and then adding the primary mapped type Date: Wed, 19 Dec 2001 23:21:41 +0000 From: Simon Nash Organization: IBM X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; I) X-Accept-Language: en MIME-Version: 1.0 To: andyp@bea.com CC: java2idl-rtf@omg.org, java-rtf@omg.org, orb_revision@omg.org Subject: [Fwd: Issue 3151 proposed resolution] Content-Type: multipart/mixed; boundary="------------054FE11052F256C23E64A21D" X-UIDL: 2G2!!nY2!!~:P!!W$`d9 Andy, Here's my proposal for issue 3151. Try not to be put off by the length. It's actually not too complicated, just very well documented :-) I did receive a comment that using a proprietary mechanism to identify a partner ORB that supports the new stream format is not a good idea, and adding a new standard service context to advertise this support would be better. I'm happy to make this change. Any feedback would be much appreciated. Simon -- Simon C Nash, Chief Technical Officer, IBM Java Technology Tel. +44-1962-815156 Fax +44-1962-818999 Hursley, England Internet: nash@hursley.ibm.com Lotus Notes: Simon Nash@ibmgbX-Mozilla-Status2: 00000000 Message-ID: <3BC9F23D.162C4F97@hursley.ibm.com> Date: Sun, 14 Oct 2001 21:14:53 +0100 From: Simon Nash Organization: IBM X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; I) X-Accept-Language: en MIME-Version: 1.0 To: Mary Leland , Vijay Natarajan , Stefan Bauer , Jeff Mischkinsky , Yoshitaka Honishi , Bob Scheifler CC: java2idl-rtf@omg.org Subject: Issue 3151 proposed resolution Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Since issue 3151 has been deferred by the last two RTFs, I am very much hoping that we will be able to resolve it this time. Here's an example of the case that doesn't work: public class A implements java.io.Serializable { // any data } public class B extends A { // any data private void writeObject(java.io.ObjectOutputStream stream) { // any code } private void readObject(java.io.ObjectInputStream stream) { // any code } } public class C extends B { // any data } At present, an instance of C is marshalled as C-header A-data B-data C-data C-endtag where all the data is written and read by the ValueHandler and is chunked according to the valuetype chunking rules. Suppose some change to the writeObject() implementation in class B cause secondary valuetype is the same as the name of the primary IDL custom value type to which the RMI/IDL value type was mapped. The secondary valuetype has no inheritance, data members, methods, or initializers. It has a #pragma ID specifying a repository ID formed by taking the repository ID of the primary custom valuetype and prefixing the Java package name with "org.omg.customRMI.". The secondary custom valuetype represents the enclosure of writeObject data that is wriitten to the serialization stream when the primary custom valuetype or any of its subclasses is serialized using format version 2, as described in item 1d of section 1.4.10. For IDL custom marshaling and unmarshaling of the primary mapped IDL valuetype, the marshal() and unmarshal() methods can call write_Value() and read_Value() to write and read the nested valuetype enclosure. This will cause the marshal() and unmarshal() methods of the secondary mapped IDL valuetype to be called to write and read the custom serialized data. 3. In existing section 1.3.5.9, add the following to the generated IDL: module org { module omg { module customRMI { module alpha { module bravo { custom valuetype Kangaroo {}; #pragma ID Kangaroo "RMI:org.omg.customRMI.alpha.bravo.Kangaroo:87654321ABCDEF01:9876543210FEDCBA" }; }; }; }; }; 4. In section 1.4.10, change paragraph 1a to read: a. octet - Format version. 1 or 2. 5. In the same section, change paragraph 1d to read: d. (optional) Additional data written by writeObject, encoded as specified below. For format version 1, the data is written "as is". For format version 2, the data is enclosed within a CDR custom valuetype with no codebase and repid "RMI:org.omg.custom." where is the fully-qualified name of the class whose writeObject method is being invoked. 6. In the same section, at the end of subsection 1, add the following: For interoperabilty using GIOP 1.2, format version 1 is required. 7. Add a new section 1.5.1.3 "ValueStream", renumbering the existing section 1.5.1.3 and subsequent sections: The interface org.omg.CORBA.portable.ValueStream defines methods that allow serialization of custom-marshaled RMI/IDL objects to and from GIOP streams. // Java package org.omg.CORBA.portable; public interface ValueStream { void start_value(java.lang.String rep_id); void end_value(); } 8. In existing section 1.5.1.3, add a new paragraph at the end of the "Execution Model for Serialization" section as follows: If the ORB supports custom serialization format 2 as described in item 1a of section 1.4.10, then the ORB streams passed to the readValue and writeValue methods of the ValueHandler object must implement the ValueStream interface. For output streams, the start_value() method ends any currently open chunk, writes a valuetype header for a nested custom valuetype (with a null codebase and the specified repository ID), and increments the valuetype nesting depth. The end_value() method ends any currently open chunk, writes the end tag for the nested custom valuetype, and decrements the valuetype nesting depth. For input streams, the start_value() method reads a valuetype header for a nested custom valuetype and increments the valuetype nesting depth. The end_value() method reads the end tag for the nested custom valuetype and decrements the valuetype nesting depth. 9. In existing section 1.5.1.3, add a new paragraph at the end of the "Value Marshaling" section as follows: If the ValueHandler supports custom serialization format 2 as described in item 1a of section 1.4.10, then the ValueHandler object must call the ValueStream.start_value() and ValueStream.end_value() methods of the ORB stream before and after writing the data specified by item 1d of section 1.4.10. The rep_id string passed to the start_value() method must be "RMI:org.omg.custom.::" where is the fully-qualified name of the class whose writeObject method is being invoked and and are the class's hashcode and SUID. 10. In existing section 1.5.1.3, add a new paragraph at the end of the "Value Unmarshaling" section as follows: If the ValueHandler supports custom serialization format 2 as described in item 1a of section 1.4.10, then the ValueHandler object must call the ValueStream.start_value() and ValueStream.end_value() methods of the ORB stream before and after reading the data specified by item 1d of section 1.4.10. The rep_id string passed to the start_value() method must be null. I apologize for the length and complexity of this proposal. As Einstein is famously quoted as saying: "Things should be as simple as possible, but no simpler." I believe this proposal meets that test, but I'm happy to discuss any alternatives that RTF members would like to put forward. The best way forward might be to schedule a conference call to go over the issue and my proposal in detail and discuss any alternative ideas for resolving this issue. Your feedback would be much appreciated. Simon -- Simon C Nash, Chief Technical Officer, IBM Java Technology Tel. +44-1962-815156 Fax +44-1962-818999 Hursley, England Internet: nash@hursley.ibm.com Lotus Notes: Simon Nash@ibmgb s more B-data to be written, while everything else stays the same. When the old B code unmarshals a new B instance, it will not fully consume all the B-data and this will cause the unmarshalling of the C-data to be incorrect since the stream will be out of sync. The problem is caused by a deficiency in the stream encoding for custom serialized data. In the example above, the existing format does not identify the boundary between the B-data and the C-data, so it is impossible for the unmarshalling code in the ValueHandler to skip over any extraneous B-data before it starts to read the C-data. Having carefully considered a number of possible solutions to this problem, I think Bob Scheifler's proposal of enclosing the B-data in a nested valuetype is the best approach. The repid for the nested valuetype must be specified by the Java to IDL mapping so that it can be read by IDL custom unmarshalling code in other languages. I propose that this repid be "RMI:org.omg.custom.::" where is the fully-qualified name of the class whose writeObject method is being invoked and and are the class's hashcode and SUID. The other language will have to provide a suitable custom valuetype implementation for the nested valuetype and may have to provide a value factory that maps the standard repid to this implementation. For Java unmarshalling, the ValueHandler will understand the special repid and process its contents as RMI-IIOP custom serialized data. Since valuetype headers and end tags are written by the ORB's OutputStream and not by the ValueHandler, it will be necessary to add new APIs to the OutputStream for the ValueHandler to call when marshalling custom valuetype data. These new APIs will write the valuetype header and end tag for the nested valuetype, keep track of the valuetype nesting level, and take care of chunking for the nested valuetype. Rather than adding yet another level of OutputStream subclassing, I propose adding the new APIs to an interface called org.omg.CORBA.portable.ValueStream which will be implemented by the output stream passed to the ValueHandler. The new interface is defined as follows: public interface ValueStream { void start_value(java.lang.String rep_id); void end_value(); } When the ValueHandler encounters a class with custom serialized data, it calls the start_value() method on the output stream that it was given, passing the special custom repid for the custom-serialized class. The output stream ends the currently open chunk, writes a valuetype header for the nested custom valuetype, and increments the valuetype nesting depth. It then returns to the ValueHandler, which calls writeObject() to write the custom serialized data. When writeObject() returns, the ValueHandler calls end_value() on the stream to end the current chunk, write the end tag, and decrement the valuetype nesting depth. When unmarshalling a class with custom serialized data, the ValueHandler needs to call the ORB's input stream to consume the valuetype header and end tag. The ValueStream interface can also be used for this purpose. The ValueHandler calls the start_value() method on the input stream, passing null as the repid. The input stream reads the nested valuetype header and increments the valuetype nesting depth, then returns to the ValueHandler. The ValueHandler reads the custom serialized data, then calls end_value() which consumes the end tag and decrements the valuetype nesting depth. For languages other than Java, the nested valuetype must be handled explicitly by calling write_Value() and read_Value() within the implementations of the marshal() and unmarshal() methods. To assist in the implementation of these methods, a new IDL valuetype is generated by the Java to IDL mapping. In the example above, this would be: module org { module omg { module customRMI { custom valuetype B {}; #pragma ID B "RMI:org.omg.customRMI.B:1234567890ABCDEF:FEDCBA0987654321" }; }; }; This would be implemented in C++ (for example) with marshal() and unmarshal() methods that read and write the appropriate data. The new stream format would be identified by a format version of 2 (currently 1). Because GIOP 1.2 uses format version 1, support for format version 2 cannot be made mandatory until GIOP 1.3. However, format version 2 can still be defined as an option for ORBs to support if they know they are talking to a partner ORB that understands this format. This could be via some proprietary means such as a vendor-specific service context or a configuration property. Here's a straw-man proposal for the changes needed to the Java to IDL mapping spec, document number ptc/2001-06-15: 1. In section 1.3.5.6, add the following at the end of the second paragraph: An additional IDL custom valuetype in the module ::org::omg::customRMI is also generated to assist with marshaling and unmarshaling instances of the class. See section 1.3.5.8 for details. 2. Add a new section 1.3.5.8 "Secondary custom valuetype", renumbering the existing section 1.3.5.8 and subsequent sections: In addition to the primary mapping described above, an RMI/IDL value type containing a writeObject method is mapped to a secondary IDL custom valuetype. The module name for this valuetype is formed by taking the ::org::omg::customRMI prefix and then adding the primary mapped type Date: Wed, 19 Dec 2001 23:21:41 +0000 From: Simon Nash Organization: IBM X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; I) X-Accept-Language: en MIME-Version: 1.0 To: andyp@bea.com CC: java2idl-rtf@omg.org, java-rtf@omg.org, orb_revision@omg.org Subject: [Fwd: Issue 3151 proposed resolution] Content-Type: multipart/mixed; boundary="------------054FE11052F256C23E64A21D" X-UIDL: 2G2!!nY2!!~:P!!W$`d9 Andy, Here's my proposal for issue 3151. Try not to be put off by the length. It's actually not too complicated, just very well documented :-) I did receive a comment that using a proprietary mechanism to identify a partner ORB that supports the new stream format is not a good idea, and adding a new standard service context to advertise this support would be better. I'm happy to make this change. Any feedback would be much appreciated. Simon -- Simon C Nash, Chief Technical Officer, IBM Java Technology Tel. +44-1962-815156 Fax +44-1962-818999 Hursley, England Internet: nash@hursley.ibm.com Lotus Notes: Simon Nash@ibmgbX-Mozilla-Status2: 00000000 Message-ID: <3BC9F23D.162C4F97@hursley.ibm.com> Date: Sun, 14 Oct 2001 21:14:53 +0100 From: Simon Nash Organization: IBM X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; I) X-Accept-Language: en MIME-Version: 1.0 To: Mary Leland , Vijay Natarajan , Stefan Bauer , Jeff Mischkinsky , Yoshitaka Honishi , Bob Scheifler CC: java2idl-rtf@omg.org Subject: Issue 3151 proposed resolution Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Since issue 3151 has been deferred by the last two RTFs, I am very much hoping that we will be able to resolve it this time. Here's an example of the case that doesn't work: public class A implements java.io.Serializable { // any data } public class B extends A { // any data private void writeObject(java.io.ObjectOutputStream stream) { // any code } private void readObject(java.io.ObjectInputStream stream) { // any code } } public class C extends B { // any data } At present, an instance of C is marshalled as C-header A-data B-data C-data C-endtag where all the data is written and read by the ValueHandler and is chunked according to the valuetype chunking rules. Suppose some change to the writeObject() implementation in class B cause secondary valuetype is the same as the name of the primary IDL custom value type to which the RMI/IDL value type was mapped. The secondary valuetype has no inheritance, data members, methods, or initializers. It has a #pragma ID specifying a repository ID formed by taking the repository ID of the primary custom valuetype and prefixing the Java package name with "org.omg.customRMI.". The secondary custom valuetype represents the enclosure of writeObject data that is wriitten to the serialization stream when the primary custom valuetype or any of its subclasses is serialized using format version 2, as described in item 1d of section 1.4.10. For IDL custom marshaling and unmarshaling of the primary mapped IDL valuetype, the marshal() and unmarshal() methods can call write_Value() and read_Value() to write and read the nested valuetype enclosure. This will cause the marshal() and unmarshal() methods of the secondary mapped IDL valuetype to be called to write and read the custom serialized data. 3. In existing section 1.3.5.9, add the following to the generated IDL: module org { module omg { module customRMI { module alpha { module bravo { custom valuetype Kangaroo {}; #pragma ID Kangaroo "RMI:org.omg.customRMI.alpha.bravo.Kangaroo:87654321ABCDEF01:9876543210FEDCBA" }; }; }; }; }; 4. In section 1.4.10, change paragraph 1a to read: a. octet - Format version. 1 or 2. 5. In the same section, change paragraph 1d to read: d. (optional) Additional data written by writeObject, encoded as specified below. For format version 1, the data is written "as is". For format version 2, the data is enclosed within a CDR custom valuetype with no codebase and repid "RMI:org.omg.custom." where is the fully-qualified name of the class whose writeObject method is being invoked. 6. In the same section, at the end of subsection 1, add the following: For interoperabilty using GIOP 1.2, format version 1 is required. 7. Add a new section 1.5.1.3 "ValueStream", renumbering the existing section 1.5.1.3 and subsequent sections: The interface org.omg.CORBA.portable.ValueStream defines methods that allow serialization of custom-marshaled RMI/IDL objects to and from GIOP streams. // Java package org.omg.CORBA.portable; public interface ValueStream { void start_value(java.lang.String rep_id); void end_value(); } 8. In existing section 1.5.1.3, add a new paragraph at the end of the "Execution Model for Serialization" section as follows: If the ORB supports custom serialization format 2 as described in item 1a of section 1.4.10, then the ORB streams passed to the readValue and writeValue methods of the ValueHandler object must implement the ValueStream interface. For output streams, the start_value() method ends any currently open chunk, writes a valuetype header for a nested custom valuetype (with a null codebase and the specified repository ID), and increments the valuetype nesting depth. The end_value() method ends any currently open chunk, writes the end tag for the nested custom valuetype, and decrements the valuetype nesting depth. For input streams, the start_value() method reads a valuetype header for a nested custom valuetype and increments the valuetype nesting depth. The end_value() method reads the end tag for the nested custom valuetype and decrements the valuetype nesting depth. 9. In existing section 1.5.1.3, add a new paragraph at the end of the "Value Marshaling" section as follows: If the ValueHandler supports custom serialization format 2 as described in item 1a of section 1.4.10, then the ValueHandler object must call the ValueStream.start_value() and ValueStream.end_value() methods of the ORB stream before and after writing the data specified by item 1d of section 1.4.10. The rep_id string passed to the start_value() method must be "RMI:org.omg.custom.::" where is the fully-qualified name of the class whose writeObject method is being invoked and and are the class's hashcode and SUID. 10. In existing section 1.5.1.3, add a new paragraph at the end of the "Value Unmarshaling" section as follows: If the ValueHandler supports custom serialization format 2 as described in item 1a of section 1.4.10, then the ValueHandler object must call the ValueStream.start_value() and ValueStream.end_value() methods of the ORB stream before and after reading the data specified by item 1d of section 1.4.10. The rep_id string passed to the start_value() method must be null. I apologize for the length and complexity of this proposal. As Einstein is famously quoted as saying: "Things should be as simple as possible, but no simpler." I believe this proposal meets that test, but I'm happy to discuss any alternatives that RTF members would like to put forward. The best way forward might be to schedule a conference call to go over the issue and my proposal in detail and discuss any alternative ideas for resolving this issue. Your feedback would be much appreciated. Simon -- Simon C Nash, Chief Technical Officer, IBM Java Technology Tel. +44-1962-815156 Fax +44-1962-818999 Hursley, England Internet: nash@hursley.ibm.com Lotus Notes: Simon Nash@ibmgb s more B-data to be written, while everything else stays the same. When the old B code unmarshals a new B instance, it will not fully consume all the B-data and this will cause the unmarshalling of the C-data to be incorrect since the stream will be out of sync. The problem is caused by a deficiency in the stream encoding for custom serialized data. In the example above, the existing format does not identify the boundary between the B-data and the C-data, so it is impossible for the unmarshalling code in the ValueHandler to skip over any extraneous B-data before it starts to read the C-data. Having carefully considered a number of possible solutions to this problem, I think Bob Scheifler's proposal of enclosing the B-data in a nested valuetype is the best approach. The repid for the nested valuetype must be specified by the Java to IDL mapping so that it can be read by IDL custom unmarshalling code in other languages. I propose that this repid be "RMI:org.omg.custom.::" where is the fully-qualified name of the class whose writeObject method is being invoked and and are the class's hashcode and SUID. The other language will have to provide a suitable custom valuetype implementation for the nested valuetype and may have to provide a value factory that maps the standard repid to this implementation. For Java unmarshalling, the ValueHandler will understand the special repid and process its contents as RMI-IIOP custom serialized data. Since valuetype headers and end tags are written by the ORB's OutputStream and not by the ValueHandler, it will be necessary to add new APIs to the OutputStream for the ValueHandler to call when marshalling custom valuetype data. These new APIs will write the valuetype header and end tag for the nested valuetype, keep track of the valuetype nesting level, and take care of chunking for the nested valuetype. Rather than adding yet another level of OutputStream subclassing, I propose adding the new APIs to an interface called org.omg.CORBA.portable.ValueStream which will be implemented by the output stream passed to the ValueHandler. The new interface is defined as follows: public interface ValueStream { void start_value(java.lang.String rep_id); void end_value(); } When the ValueHandler encounters a class with custom serialized data, it calls the start_value() method on the output stream that it was given, passing the special custom repid for the custom-serialized class. The output stream ends the currently open chunk, writes a valuetype header for the nested custom valuetype, and increments the valuetype nesting depth. It then returns to the ValueHandler, which calls writeObject() to write the custom serialized data. When writeObject() returns, the ValueHandler calls end_value() on the stream to end the current chunk, write the end tag, and decrement the valuetype nesting depth. When unmarshalling a class with custom serialized data, the ValueHandler needs to call the ORB's input stream to consume the valuetype header and end tag. The ValueStream interface can also be used for this purpose. The ValueHandler calls the start_value() method on the input stream, passing null as the repid. The input stream reads the nested valuetype header and increments the valuetype nesting depth, then returns to the ValueHandler. The ValueHandler reads the custom serialized data, then calls end_value() which consumes the end tag and decrements the valuetype nesting depth. For languages other than Java, the nested valuetype must be handled explicitly by calling write_Value() and read_Value() within the implementations of the marshal() and unmarshal() methods. To assist in the implementation of these methods, a new IDL valuetype is generated by the Java to IDL mapping. In the example above, this would be: module org { module omg { module customRMI { custom valuetype B {}; #pragma ID B "RMI:org.omg.customRMI.B:1234567890ABCDEF:FEDCBA0987654321" }; }; }; This would be implemented in C++ (for example) with marshal() and unmarshal() methods that read and write the appropriate data. The new stream format would be identified by a format version of 2 (currently 1). Because GIOP 1.2 uses format version 1, support for format version 2 cannot be made mandatory until GIOP 1.3. However, format version 2 can still be defined as an option for ORBs to support if they know they are talking to a partner ORB that understands this format. This could be via some proprietary means such as a vendor-specific service context or a configuration property. Here's a straw-man proposal for the changes needed to the Java to IDL mapping spec, document number ptc/2001-06-15: 1. In section 1.3.5.6, add the following at the end of the second paragraph: An additional IDL custom valuetype in the module ::org::omg::customRMI is also generated to assist with marshaling and unmarshaling instances of the class. See section 1.3.5.8 for details. 2. Add a new section 1.3.5.8 "Secondary custom valuetype", renumbering the existing section 1.3.5.8 and subsequent sections: In addition to the primary mapping described above, an RMI/IDL value type containing a writeObject method is mapped to a secondary IDL custom valuetype. The module name for this valuetype is formed by taking the ::org::omg::customRMI prefix and then adding the primary mapped type X-Sender: andyp@san-francisco.beasys.com X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Date: Wed, 19 Dec 2001 16:17:32 -0800 To: Simon Nash From: Andy Piper Subject: Re: [Fwd: Issue 3151 proposed resolution] Cc: java2idl-rtf@omg.org, java-rtf@omg.org, orb_revision@omg.org In-Reply-To: <3C212105.ECE4A6E8@hursley.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-UIDL: "Did9_X3!!h-[d9/5jd9 Hi Simon At 11:21 PM 12/19/2001 +0000, Simon Nash wrote: Here's my proposal for issue 3151. Try not to be put off by the length. It's actually not too complicated, just very well documented :-) I like this proposal - I have a few comments. 1. What about encapsulations? These are potentially marshaled without reference to a target Orb. Most of the language says that you have to assume GIOP 1.2 encoding so I am assuming you would have to assume version 1 streams inside encapsulations (or version 2 for GIOP 1.3). If so you would need to doc this. 2. I agree a standard service context for identifying version info would be good. Would this be per-request or for the duration of a conversation? Could you alternatively use some of the value tag bits? 3. This only happens for custom-marshaled data I assume. I.e. standard serialized data is as now. 4. I don't like the overloading of the ValueStream APIs. The semantics depend on the context and implementation which is bad practice IMO. You should have one for input and one for output. Thanks andy Date: Thu, 20 Dec 2001 15:16:10 +0000 From: Simon Nash Organization: IBM X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; I) X-Accept-Language: en MIME-Version: 1.0 To: Andy Piper CC: java2idl-rtf@omg.org, java-rtf@omg.org, orb_revision@omg.org Subject: Re: [Fwd: Issue 3151 proposed resolution] References: <4.3.2.7.2.20011219161018.00b84618@san-francisco.beasys.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii X-UIDL: "!Oe9K0/e9Hf3e99QJ!! Andy, Thanks for looking at this, and for the positive response. My replies to your comments are inline below. Simon Andy Piper wrote: > > Hi Simon > > At 11:21 PM 12/19/2001 +0000, Simon Nash wrote: > >Here's my proposal for issue 3151. Try not to be put off by the length. > >It's actually not too complicated, just very well documented :-) > > I like this proposal - I have a few comments. > > 1. What about encapsulations? These are potentially marshaled without > reference to a target Orb. Most of the language says that you have to > assume GIOP 1.2 encoding so I am assuming you would have to assume version > 1 streams inside encapsulations (or version 2 for GIOP 1.3). If so you > would need to doc this. > I think the only place where an RMI/IDL value type can appear in an encapsulation is the UnknownExceptionInfo service context. When marshalling this, the server would know whether the client ORB has previously advertised (e.g., by means of the new service context) that it supports version 2 streams, and the server could use stream format 2 in such cases. Also, if the request is GIOP 1.3, the reply would use stream format 2. In all other cases the server would have to use stream format 1 for UnknownExceptionInfo. If we decide to create a new flavour of this service context to fix issue 4795, then we could specify that the new context uses stream format 2, and the current context uses stream format 1. This seems like a cleaner solution. Do you think there are any other encapsulations that could contain RMI-IIOP valuetypes? If so, I agree that the stream format should be 1 for GIOP 1.2 and 2 for GIOP 1.3. > 2. I agree a standard service context for identifying version info would be > good. Would this be per-request or for the duration of a conversation? > Could you alternatively use some of the value tag bits? > I think a service context would have to be per-request, at least when sent by a server, since otherwise there is no way for the server to know for sure that the client has received it. This also applies to the SendingContextRunTime service context when sent in this direction. The only way round this is to add a new "ack" service context that the client can send back to the server to acknowledge that it has received and processed the service context sent by the server. For each connection, the server would need to keep transmitting the RMIStreamFormat service context until it received the "ack". This problem does not apply to service contexts sent only from client to server, such as CodeSets. These can be acknowledged by the server sending back a GIOP reply message for the GIOP request carrying the service context. The GIOP reply message serves as the "ack". The same approach can be used for SendingContextRunTime service contexts sent from client to server. Using value tag bits isn't reliable since there is no guarantee that any valuetypes will have been transmitted previously from receiver to sender. The sender must know that the receiver understands stream format 2 before using it to marshal a custom valuetype. In an "in doubt" case, it can't send a custom valuetype with a tag bit set to indicate format 2 in the hope that the receiver will be able to handle it. > 3. This only happens for custom-marshaled data I assume. I.e. standard > serialized data is as now. > Yes, that's correct. > 4. I don't like the overloading of the ValueStream APIs. The semantics > depend on the context and implementation which is bad practice IMO. You > should have one for input and one for output. > Are you suggesting separate interfaces for input and output, or keeping a single ValueStream interface and having separate methods? Simon -- Simon C Nash, Chief Technical Officer, IBM Java Technology Tel. +44-1962-815156 Fax +44-1962-818999 Hursley, England Internet: nash@hursley.ibm.com Lotus Notes: Simon Nash@ibmgb Date: Sun, 23 Dec 2001 15:25:43 +0000 From: Simon Nash Organization: IBM X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; I) X-Accept-Language: en MIME-Version: 1.0 To: java2idl-rtf@omg.org CC: everett.anderson@Sun.COM, andyp@bea.com, interop@omg.org Subject: Issue 3151 and GIOP 1.3 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii X-UIDL: mdl!!BY RMICustomStreamFormat identifies a CDR encapsulation of the RMICustom::StreamFormat defined in the Java to IDL mapping specification (formal/xx-xx-xx), section 1.4.11. 13. In section 13.9, add a row to the table for feature "RMICustomStreamFormat service context" showing that it is optional for version 1.2. 14. In section A.2, add a row to the table for ServiceId tag value "RMICustomStreamFormat = 15" defined in the Java to IDL mapping specification (formal/xx-xx-xx), section 1.4.11. Simon -- Simon C Nash, Chief Technical Officer, IBM Java Technology Tel. +44-1962-815156 Fax +44-1962-818999 Hursley, England Internet: nash@hursley.ibm.com Lotus Notes: Simon Nash@ibmgb [d9Od^d9[Fod9 It is my understanding that GIOP 1.3 will be adopted soon and will differ from GIOP 1.2 only in minor ways, e.g., by adding one or two operations to CORBA::Object. If this is correct, then we could consider tying the fix for issue 3151 to GIOP 1.3 rather than adding a service context to allow the new stream format to be used with GIOP 1.2. This change would simplify the proposal. I'd appreciate any comments on this, as I'm in the process of writing up a proposed resolution for voting. Thanks. Simon -- Simon C Nash, Chief Technical Officer, IBM Java Technology Tel. +44-1962-815156 Fax +44-1962-818999 Hursley, England Internet: nash@hursley.ibm.com Lotus Notes: Simon Nash@ibmgb Date: Mon, 31 Dec 2001 21:52:18 +0000 From: Simon Nash Organization: IBM X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; I) X-Accept-Language: en MIME-Version: 1.0 To: Jishnu Mukerji , Vijay Natarajan , Harold Carr , Jeff Mischkinsky , Yoshitaka Honishi , Andy Piper , Xudong Chen , java2idl-rtf@omg.org Subject: Issue 3151 proposed resolution Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=iso-8859-1 X-UIDL: of7!!Z""!!)Kp!!#??!! Status: RO Here's a revised proposal for issue 3151, containing changes in response to feedback on the original proposal. Since I have received no objections to the substance of this proposal (which was sent out in October and again in December), I intend to put up this revised proposal for vote shortly. Issue 3151 has been deferred by the last two RTFs, but needs to be resolved in order to fix some cases of type evolution that don't currently work with RMI over IIOP. Here's an example of a case that doesn't work: public class A implements java.io.Serializable { // any data } public class B extends A { // any data private void writeObject(java.io.ObjectOutputStream stream) { // any code } private void readObject(java.io.ObjectInputStream stream) { // any code } } public class C extends B { // any data } At present, an instance of C is marshalled as C-header A-data B-data C-data C-endtag where all the data is written and read by the ValueHandler and is chunked according to the valuetype chunking rules. Suppose some change to the writeObject() implementation in class B cause secondary valuetype is the same as the name of the primary IDL custom value type to which the RMI/IDL value type was mapped. The secondary valuetype has no inheritance, data members, methods, or initializers. It has a #pragma ID specifying a repository ID formed by taking the repository ID of the primary custom valuetype and prefixing the Java package name with "org.omg.customRMI.". The secondary custom valuetype represents the enclosure of writeObject data that is wriitten to the serialization stream when the primary custom valuetype or any of its subclasses is serialized using format version 2, as described in item 1d of section 1.4.10. For IDL custom marshaling and unmarshaling of the primary mapped IDL valuetype, the marshal() and unmarshal() methods can call write_Value() and read_Value() to write and read the nested valuetype enclosure. This will cause the marshal() and unmarshal() methods of the secondary mapped IDL valuetype to be called to write and read the custom serialized data. 3. In existing section 1.3.5.9, add the following to the generated IDL: module org { module omg { module customRMI { module alpha { module bravo { custom valuetype Kangaroo {}; #pragma ID Kangaroo "RMI:org.omg.customRMI.alpha.bravo.Kangaroo:87654321ABCDEF01:9876543210FEDCBA" }; }; }; }; }; 4. In section 1.4.10, change paragraph 1a to read: a. octet - Format version. 1 or 2. 5. In the same section, change paragraph 1d to read: d. (optional) Additional data written by writeObject, encoded as specified below. For format version 1, the data is written "as is". For format version 2, the data is enclosed within a CDR custom valuetype with no codebase and repid "RMI:org.omg.custom." where is the fully-qualified name of the class whose writeObject method is being invoked. 6. In the same section, at the end of subsection 1, add the following: For interoperability using GIOP 1.2, format version 1 must be sent, except where the RMICustomStreamFormat service context (see section 1.4.11) has previously been received on the same connection. The same format version rules apply to RMI custom data sent within an encapsulation as part of the UnknownExceptionInfo service context. This is the only case where RMI custom data can appear as part of an encapsulation. 7. Add a new section 1.4.11 "RMICustomStreamFormat Service Context", renumbering the existing section 1.4.11 and subsequent sections: The RMICustomStreamFormat service context contains a CDR encapsulation of the RMICustom::StreamFormat type, defined as follows. module RMICustom { struct StreamFormat { octet maxSerializableFormatVersion; // highest serializable format supported octet minSerializableFormatVersion; // lowest serializable format supported octet maxExternalizableFormatVersion; // highest externalizable format supported octet minExternalizableFormatVersion; // lowest externalizable format supported }; }; This service context allows ORBs to indicate that they are able to accept a higher or lower stream format version (or both) for RMI custom data than the format versions required by the GIOP level of the message used to send the valuetype containing RMI custom data. The RMICustomStreamFormat service context can be sent on any GIOP message that can carry a service context. An ORB that receives this service context is allowed to send on the same connection RMI custom data with any stream format version that is within the bounds specified by the service context. It is always permissible to send any stream format version whose support is defined as mandatory for the message GIOP level being sent. 8. Add new section 1.5.1.3 "ValueOutputStream" and 1.5.1.4 "ValueInputStream", renumbering the existing section 1.5.1.3 and subsequent sections: The interface org.omg.CORBA.portable.ValueOutputStream defines methods that allow serialization of custom-marshaled RMI/IDL objects to GIOP streams. // Java package org.omg.CORBA.portable; public interface ValueOutputStream { void start_value(java.lang.String rep_id); void end_value(); } The interface org.omg.CORBA.portable.ValueInputStream defines methods that allow deserialization of custom-marshaled RMI/IDL objects from GIOP streams. // Java package org.omg.CORBA.portable; public interface ValueInputStream { void start_value(); void end_value(); } 9. In existing section 1.5.1.3, add a new paragraph at the end of the "Execution Model for Serialization" section as follows: If the ORB supports custom serialization format 2 as described in item 1a of section 1.4.10, then the ORB streams passed to the writeValue and readValue methods of the ValueHandler object must implement the ValueOutputStream and ValueInputStream interfaces respectively. For output streams, the start_value() method ends any currently open chunk, writes a valuetype header for a nested custom valuetype (with a null codebase and the specified repository ID), and increments the valuetype nesting depth. The end_value() method ends any currently open chunk, writes the end tag for the nested custom valuetype, and decrements the valuetype nesting depth. For input streams, the start_value() method reads a valuetype header for a nested custom valuetype and increments the valuetype nesting depth. The end_value() method reads the end tag for the nested custom valuetype (after skipping any data that precedes the end tag) and decrements the valuetype nesting depth. 10. In existing section 1.5.1.3, add a new paragraph at the end of the "Value Marshaling" section as follows: If the ValueHandler supports custom serialization format 2 as described in item 1a of section 1.4.10, then the ValueHandler object must call the ValueOutputStream.start_value() and ValueOutputStream.end_value() methods of the ORB stream before and after writing the data specified by item 1d of section 1.4.10. The rep_id string passed to the start_value() method must be "RMI:org.omg.custom.::" where is the fully-qualified name of the class whose writeObject method is being invoked and and are the class's hashcode and SUID. 11. In existing section 1.5.1.3, add a new paragraph at the end of the "Value Unmarshaling" section as follows: If the ValueHandler supports custom serialization format 2 as described in item 1a of section 1.4.10, then the ValueHandler object must call the ValueInputStream.start_value() and ValueInputStream.end_value() methods of the ORB stream before and after reading the data specified by item 1d of section 1.4.10. and the changes needed to the CORBA specification, document number formal/2001-09-34: 12. In section 13.7.1, add the following ServiceId constant definition: const ServiceId RMICustomStreamFormat = 15; and the following bullet: s more B-data to be written, while everything else stays the same. When the old B code unmarshals a new B instance, it will not fully consume all the B-data and this will cause the unmarshalling of the C-data to be incorrect since the stream will be out of sync. The problem is caused by a deficiency in the stream encoding for custom serialized data. In the example above, the existing format does not identify the boundary between the B-data and the C-data, so it is impossible for the unmarshalling code in the ValueHandler to skip over any extraneous B-data before it starts to read the C-data. Having carefully considered a number of possible solutions to this problem, I think Bob Scheifler's proposal of enclosing the B-data in a nested valuetype is the best approach. The repid for the nested valuetype must be specified by the Java to IDL mapping so that it can be read by IDL custom unmarshalling code in other languages. I propose that this repid be "RMI:org.omg.custom.::" where is the fully-qualified name of the class whose writeObject method is being invoked and and are the class's hashcode and SUID. The other language will have to provide a suitable custom valuetype implementation for the nested valuetype and may have to provide a value factory that maps the standard repid to this implementation. For Java unmarshalling, the ValueHandler will understand the special repid and process its contents as RMI-IIOP custom serialized data. Since valuetype headers and end tags are written by the ORB's OutputStream and not by the ValueHandler, it will be necessary to add new APIs to the OutputStream for the ValueHandler to call when marshalling custom valuetype data. These new APIs will write the valuetype header and end tag for the nested valuetype, keep track of the valuetype nesting level, and take care of chunking for the nested valuetype. Rather than adding yet another level of OutputStream subclassing, I propose adding the new APIs to an interface called org.omg.CORBA.portable.ValueOutputStream which will be implemented by the output stream passed to the ValueHandler. The new interface is defined as follows: public interface ValueOutputStream { void start_value(java.lang.String rep_id); void end_value(); } When the ValueHandler encounters a class with custom serialized data, it calls the start_value() method on the output stream that it was given, passing the special custom repid for the custom-serialized class. The output stream ends the currently open chunk, writes a valuetype header for the nested custom valuetype, and increments the valuetype nesting depth. It then returns to the ValueHandler, which calls writeObject() to write the custom serialized data. When writeObject() returns, the ValueHandler calls end_value() on the stream to end the current chunk, write the end tag, and decrement the valuetype nesting depth. When unmarshalling a class with custom serialized data, the ValueHandler needs to call the ORB's input stream to consume the valuetype header and end tag. A new interface org.omg.CORBA.portable.ValueInputStream that defines the necessary APIs will be implemented by the input stream. This interface is defined as follows: public interface ValueInputStream { void start_value(); void end_value(); } The ValueHandler calls the start_value() method on the input stream. The input stream reads the nested valuetype header and increments the valuetype nesting depth, then returns to the ValueHandler. The ValueHandler reads the custom serialized data, then calls end_value() which consumes the end tag (after skipping any data that precedes the end tag) and decrements the valuetype nesting depth. For languages other than Java, the nested valuetype must be handled explicitly by calling write_Value() and read_Value() within the implementations of the marshal() and unmarshal() methods. To assist in the implementation of these methods, a new IDL valuetype is generated by the Java to IDL mapping. In the example above, this would be: module org { module omg { module customRMI { custom valuetype B {}; #pragma ID B "RMI:org.omg.customRMI.B:1234567890ABCDEF:FEDCBA0987654321" }; }; }; This would be implemented in C++ (for example) with marshal() and unmarshal() methods that read and write the appropriate data. The new stream format would be identified by a format version of 2 (currently 1). Because GIOP 1.2 uses format version 1, support for format version 2 cannot be made mandatory until GIOP 1.3. In order to permit ORBs to use stream format 2 as an option over GIOP 1.2, a new service context is defined as follows: module RMICustom { struct StreamFormat { octet maxSerializableFormatVersion; // highest serializable format supported octet minSerializableFormatVersion; // lowest serializable format supported octet maxExternalizableFormatVersion; // highest externalizable format supported octet minExternalizableFormatVersion; // lowest externalizable format supported }; }; together with a new ServiceId constant: const ServiceId RMICustomStreamFormat = 15; RMICustomStreamFormat identifies a CDR encapsulation of the RMICustom::StreamFormat defined above. It allows ORBs to indicate that they are able to accept a higher or lower stream format version (or both) for RMI custom data than the format required by the GIOP level of the message used to send the valuetype containing RMI custom data. The RMICustomStreamFormat service context can be sent on any GIOP message that can carry a service context. An ORB that receives this service context is allowed to send on the same connection RMI custom data with any stream format version that is within the bounds specified by the service context. It is always permissible to send any stream format version whose support is defined as mandatory for the message GIOP level being sent. RMI custom data sent within an encapsulation as part of the UnknownExceptionInfo service context must have a stream format version that is compatible with either a. an RMICustomStreamFormat service context previously received over the same connection, if such a service context has been received on this connection, or b. any stream format version whose support is mandatory for the GIOP level of the GIOP message carrying the UnknownExceptionInfo service context. This is the only case where RMI custom data can appear as part of an encapsulation. Here's a formal proposal for the changes needed to the Java to IDL mapping spec, document number ptc/2001-06-15: 1. In section 1.3.5.6, add the following at the end of the second paragraph: An additional IDL custom valuetype in the module ::org::omg::customRMI is also generated to assist with marshaling and unmarshaling instances of the class. See section 1.3.5.8 for details. 2. Add a new section 1.3.5.8 "Secondary custom valuetype", renumbering the existing section 1.3.5.8 and subsequent sections: In addition to the primary mapping described above, an RMI/IDL value type containing a writeObject method is mapped to a secondary IDL custom valuetype. The module name for this valuetype is formed by taking the ::org::omg::customRMI prefix and then adding the primary mapped type X-Sender: andyp@san-francisco.beasys.com X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Date: Thu, 10 Jan 2002 12:38:15 -0800 To: Simon Nash , Jishnu Mukerji , Vijay Natarajan , Harold Carr , Jeff Mischkinsky , Yoshitaka Honishi , Xudong Chen From: Andy Piper Subject: Re: Java to IDL 2001 RTF Vote 3 Cc: java2idl-rtf@omg.org, juergen@omg.org In-Reply-To: <3C34DF7A.32E0A64E@hursley.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-UIDL: 26f!!^cFe9WcY!!Zbm!! BEA Votes: At 10:47 PM 1/3/2002 +0000, Simon Nash wrote: There are 2 issues in this vote: 3151: Stream format problem for custom marshalled RMI types NO (but purely in relation to the version part, can we separate this out?), I think everything else is great. 4591: Behavior of writeByte, writeChar, writeBytes, writeChars YES andy Date: Thu, 10 Jan 2002 21:39:02 +0000 From: Simon Nash Organization: IBM X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; I) X-Accept-Language: en MIME-Version: 1.0 To: Andy Piper CC: Jishnu Mukerji , Vijay Natarajan , Harold Carr , Yoshitaka Honishi , Xudong Chen , java2idl-rtf@omg.org Subject: Re: Java to IDL 2001 RTF Vote 3 References: <4.3.2.7.2.20020110123419.00b535e0@san-francisco.beasys.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii X-UIDL: Jg8!!_R'e9:?'e9WUh!! Andy, Andy Piper wrote: > > BEA Votes: > > At 10:47 PM 1/3/2002 +0000, Simon Nash wrote: > >There are 2 issues in this vote: > > 3151: Stream format problem for custom marshalled RMI types > > NO (but purely in relation to the version part, can we separate this out?), > I think everything else is great. > I'd be happy to take this out and say that the fix requires GIOP 1.3. Would that be OK with you? Simon -- Simon C Nash, Chief Technical Officer, IBM Java Technology Tel. +44-1962-815156 Fax +44-1962-818999 Hursley, England Internet: nash@hursley.ibm.com Lotus Notes: Simon Nash@ibmgb X-Sender: andyp@san-francisco.beasys.com X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Date: Thu, 10 Jan 2002 14:47:18 -0800 To: Simon Nash From: Andy Piper Subject: Re: Java to IDL 2001 RTF Vote 3 Cc: Jishnu Mukerji , Vijay Natarajan , Harold Carr , Yoshitaka Honishi , Xudong Chen , java2idl-rtf@omg.org In-Reply-To: <3C3E09F6.FE68AB4@hursley.ibm.com> References: <4.3.2.7.2.20020110123419.00b535e0@san-francisco.beasys.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-UIDL: )T0!!df;e9OR$!!Y~Ce9 Status: RO At 09:39 PM 1/10/2002 +0000, Simon Nash wrote: I'd be happy to take this out and say that the fix requires GIOP 1.3. Would that be OK with you? I think that would be fine, we could then raise another issue for pursuing interop with previous versions. andy Date: Thu, 10 Jan 2002 15:20:50 -0800 From: Harold Carr X-Mailer: Mozilla 4.51 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: Simon Nash CC: Jishnu Mukerji , Vijay Natarajan , Jeff Mischkinsky , Yoshitaka Honishi , Andy Piper , Xudong Chen , java2idl-rtf@omg.org, juergen@omg.org Subject: Re: Java to IDL 2001 RTF Vote 3 References: <3C34DF7A.32E0A64E@hursley.ibm.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii X-UIDL: CZ!"!=i/!!Im8e9#AV!! Status: RO Sun votes as follows: YES: 4591: Behavior of writeByte, writeChar, writeBytes, writeChars ABSTAIN: 3151: Stream format problem for custom marshalled RMI types The initial proposal included versioning. One voter did not want to include versioning so the chair said he would remove it. But that makes it a different resolution than others have already voted on. I think this round of votes cannot now include 3151. If changes are made to the resolution than the new resolution(s) should be put up for a vote in the next vote. Questions on 3151: 1. Why is anyone against the versioning in the proposal? 2. Why just solve it going forward when versioning can solve it going backward as well? 3. Won't making the versioning dependent on GIOP 1.3 slow down the solution consideraby compared to including a versioning scheme in a Java-to-IDL solution? 4. This seems to be involving solutions to 4795 also, right? Thanks, Harold Date: Thu, 10 Jan 2002 18:19:51 -0500 From: Jishnu Mukerji Organization: Hewlett-Packard Company X-Mailer: Mozilla 4.78 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: Simon Nash Cc: java2idl-rtf@omg.org, everett.anderson@Sun.COM, andyp@bea.com, interop@omg.org Subject: Re: Issue 3151 and GIOP 1.3 References: <3C25F777.B7F351B1@hursley.ibm.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii X-UIDL: 4ZJe9/Dg!!ABGe9on6e9 Status: RO Simon Nash wrote: > > It is my understanding that GIOP 1.3 will be adopted soon and will differ > from GIOP 1.2 only in minor ways, e.g., by adding one or two operations to > CORBA::Object. If this is correct, then we could consider tying the fix > for issue 3151 to GIOP 1.3 rather than adding a service context to allow > the new stream format to be used with GIOP 1.2. This change would simplify > the proposal. > > I'd appreciate any comments on this, as I'm in the process of writing up a > proposed resolution for voting. Thanks. Looking back at the history of GIOP 1.3 discussion, it seems to me that we do need to delay GIOP 1.3 at least until the Components and Firewall FTF complete their tasks. Given that Firewall is completing final submissions at this meeting, I don't see how this can be achieved before the middle of the year at the earliest (Orlando meeting?) We should then take at least one meeting cycle of serious work in the Interop RTF to figure out how to throw these disperate pieces of changes to GIOP coming in from various directions need to be aligned together, just for everyones sanity's sake. Here are the significant components going into various GIOP changes: 1. Components spec (which is ready now) adds _component (or some such). They did punt on the other thing which should probably be resurrected in the Interop RTF at some point. Anyway either one of those requires a minor number rev. 2. Firewall spec, adds random significant things to GIOP. This will probably not be finalized until say 3Q02. 3. A few proposals that are floating around in Core, Java-IDL etc RTFs add bits and pieces to GIOP that should require revving minor number. Some of these could be ready late 1Q/early 2Q. 4. Some others that I may have missed (can anyone think of anything else? Now, here is the rub. If we delay GIOP 1.3 all the way to 3Q, that means that in effect interoperable Components specification will not be available until 3Q02, which may not be acceptable to those that plan to use the Components spec. Also changes that happen as late as 3/4Q 02 will also miss the next J2EE bus, as far as I can tell. So we need to come up with a plan that minimizes the pain for everyone and maximizes the returns. This may require two minor revisions of GIOP suitably timed through 2002. But this requires some serious headbashing and I propose that we do some serious brainstorming on this matter and come up with a plan by the end of the Anaheim meeting and follow through oin it with due diligence. We need to decide: i. How many minor revs and what timeframes. ii. Contents of each minor rev - perhaps do a triage of the infamous wishlist (actually more like a lets punt list) and see what items from it can be realisitically handled in the given timeframes. iii. Keep in mind that it is inappropriate to delay already adopted specs from being formally published based on speculative contents of speculative minor revisions.:-) What says all of you? Cheers, Jishnu. Sender: jon@floorboard.com Message-ID: <3C3E3079.2A0B0B85@floorboard.com> Date: Thu, 10 Jan 2002 16:23:21 -0800 From: Jonathan Biggar X-Mailer: Mozilla 4.77 [en] (X11; U; SunOS 5.7 sun4u) X-Accept-Language: en MIME-Version: 1.0 To: Jishnu Mukerji CC: Simon Nash , java2idl-rtf@omg.org, everett.anderson@Sun.COM, andyp@bea.com, interop@omg.org Subject: Re: Issue 3151 and GIOP 1.3 References: <3C25F777.B7F351B1@hursley.ibm.com> <3C3E2197.FD2DA152@hp.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii X-UIDL: ,`Me957#e9Af[d9%b8!! Status: RO Jishnu Mukerji wrote: > Looking back at the history of GIOP 1.3 discussion, it seems to me that > we do need to delay GIOP 1.3 at least until the Components and Firewall > FTF complete their tasks. Given that Firewall is completing final > submissions at this meeting, I don't see how this can be achieved before > the middle of the year at the earliest (Orlando meeting?) We should then > take at least one meeting cycle of serious work in the Interop RTF to > figure out how to throw these disperate pieces of changes to GIOP coming > in from various directions need to be aligned together, just for > everyones sanity's sake. > > Here are the significant components going into various GIOP changes: > > 1. Components spec (which is ready now) adds _component (or some such). > They did punt on the other thing which should probably be resurrected in > the Interop RTF at some point. Anyway either one of those requires a > minor number rev. > > 2. Firewall spec, adds random significant things to GIOP. This will > probably not be finalized until say 3Q02. > > 3. A few proposals that are floating around in Core, Java-IDL etc RTFs > add bits and pieces to GIOP that should require revving minor number. > Some of these could be ready late 1Q/early 2Q. > > 4. Some others that I may have missed (can anyone think of anything > else? > > Now, here is the rub. If we delay GIOP 1.3 all the way to 3Q, that means > that in effect interoperable Components specification will not be > available until 3Q02, which may not be acceptable to those that plan to > use the Components spec. Also changes that happen as late as 3/4Q 02 > will also miss the next J2EE bus, as far as I can tell. So we need to > come up with a plan that minimizes the pain for everyone and maximizes > the returns. This may require two minor revisions of GIOP suitably timed > through 2002. But this requires some serious headbashing and I propose > that we do some serious brainstorming on this matter and come up with a > plan by the end of the Anaheim meeting and follow through oin it with > due diligence. We need to decide: > > i. How many minor revs and what timeframes. > > ii. Contents of each minor rev - perhaps do a triage of the infamous > wishlist (actually more like a lets punt list) and see what items from > it can be realisitically handled in the given timeframes. > > iii. Keep in mind that it is inappropriate to delay already adopted > specs from being formally published based on speculative contents of > speculative minor revisions.:-) > > What says all of you? Sounds like a good plan. One thing I've been wondering of late, is whether it really makes sense to have separate Core & Interop RTFs. The two are separate more because of accident than a purposeful choice, and it seems to me that in a lot of cases, having two just means a lot of extra coordination since a lot of Core issues eventually need GIOP changes as well. The company representation of the two RTFs are very close to the same list, although there is considerable difference in the individuals involved. So, consider the question posed: should we look at merging the two RTFs? -- Jon Biggar Floorboard Software jon@floorboard.com jon@biggar.org From: Jeffrey Mischkinsky Message-Id: <200201110046.QAA01540@wheel.dcn.davis.ca.us> Subject: Re: Java to IDL 2001 RTF Vote 3 To: nash@hursley.ibm.com (Simon Nash) Date: Thu, 10 Jan 2002 16:46:48 -0800 (PST) Cc: jishnu_mukerji@hp.com (Jishnu Mukerji), vnatarajan@borland.com (Vijay Natarajan), harold.carr@Sun.COM (Harold Carr), jeff.mischkinsky@oracle.com (Jeff Mischkinsky), honi@fsc.fujitsu.com (Yoshitaka Honishi), andyp@bea.com (Andy Piper), xdchen@borland.com (Xudong Chen), java2idl-rtf@omg.org In-Reply-To: <3C3B192E.CC9AFC0B@hursley.ibm.com> from "Simon Nash" at Jan 08, 2002 04:07:10 PM X-Mailer: ELM [version 2.5 PL2] MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=iso-8859-1 X-UIDL: 8Ppd9Z37!!~0k!!fk'!! Status: RO ORCL votes as follows: 4591: YES 3151: NO - this seems to me to really require a change to GIOP 1.3 hence is premature at this point. I'm not convinced by the "trick" of backdooring it via a new service context, which no one who supports 1.2 is required to support--or it really is a GIOP rev. If both sides have to agree, then there is no interop anyway, and it might as well be proprietary. Also this seems like a rather significant/complex change and I haven't been convinced by the discussion that it has been fully thought through and tested (between vendors). So bottom line: i need to be convinced that it is fully baked. particularly because this is also a requirement on all J2EE vendors via EJB interop. We need to be very cautious here. jeff 'Simon Nash' writes: > > IBM votes Yes on both issues. > > Simon > > Simon Nash wrote: > > > > RTF Members: Please see the attached html file for the issues to be voted on and > > their proposed resolutions. Votes are due by 6pm US PST on January 10, 2001. > > > > There are 2 issues in this vote: > > 3151: Stream format problem for custom marshalled RMI types > > 4591: Behavior of writeByte, writeChar, writeBytes, writeChars > > > > The voting members of the Java to IDL RTF are: > > Simon Nash, IBM > > Jishnu Mukerji, HP > > Vijay Natarajan, Borland (proxy: Xudong Chen) > > Harold Carr, Sun > > Jeff Mischkinsky, Oracle > > Yoshitaka Honishi, Fujitsu > > Andy Piper, BEA > > > > Juergen: Please put the attached html file in the > > http://cgi.omg.org/pub/java2idl-rtf > > directory. Thanks. > > > > Simon > > -- > > Simon C Nash, Chief Technical Officer, IBM Java Technology > > Tel. +44-1962-815156 Fax +44-1962-818999 Hursley, England > > Internet: nash@hursley.ibm.com Lotus Notes: Simon Nash@ibmgb > > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > > > > Java to IDL 2001 RTF Vote 3 > > > > Deadline for Votes: > > 6pm, US Pacific Standard Time, January 10, 2001 > > > > Summary of Issues in this Vote > > > > Issue 3151: Stream format problem for custom marshalled RMI types > > Issue 4591: Behavior of writeByte, writeChar, writeBytes, writeChars > > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > > > Issue 3151: Stream format problem for custom marshalled RMI types > > > > Click here for this issue's archive. > > Source: International Business Machines (Mr. Simon C. Nash, mailto:%20nash@hursley.ibm.com) > > Nature: Uncategorized Issue > > Severity: > > Summary: There is a problem in the stream format for custom marshalled RMI types. The Java serialization spec requires that when a class with a writeObject method calls defaultWriteObject and then writes optional data, the receiver must be able to skip > > the optional data if the readObject does not consume it or the class is not in the receiver's hierarchy. So if the sender's hierarchy consists of serializable classes Base and Derived, both of which call defaultWriteObject and then write optional data, > > there must be a way for the unmarshaller to first skip to the end of the Base data, then skip to the end of the Derived data. With the current stream format, this is not possible. > > Proposed Resolution: > > The following example shows a case that currently doesn't work: > > > > public class A implements java.io.Serializable { > > // any data > > } > > > > public class B extends A { > > // any data > > private void writeObject(java.io.ObjectOutputStream stream) { > > // any code > > } > > private void readObject(java.io.ObjectInputStream stream) { > > // any code > > } > > } > > > > public class C extends B { > > // any data > > } > > > > At present, an instance of C is marshalled as > > > > C-header A-data B-data C-data C-endtag > > > > where all the data is written and read by the ValueHandler and is chunked according to the valuetype chunking rules. Suppose some change to the writeObject() implementation in class B cause> > members, methods, or initializers. It has a #pragma ID specifying a repository ID formed by taking the repository ID of the primary custom valuetype and prefixing the Java package name with "org.omg.customRMI.". The secondary custom valuetype > > represents the enclosure of writeObject data that is wriitten to the serialization stream when the primary custom valuetype or any of its subclasses is serialized using format version 2, as described in item 1d of section 1.4.10. For IDL custom > > marshaling and unmarshaling of the primary mapped IDL valuetype, the marshal() and unmarshal() methods can call write_Value() and read_Value() to write and read the nested valuetype enclosure. This will cause the marshal() and unmarshal() methods > > of the secondary mapped IDL valuetype to be called to write and read the custom serialized data." > > 3. In existing section 1.3.5.9, add the following to the generated IDL: > > > > module org { > > module omg { > > module customRMI { > > module alpha { > > module bravo { > > custom valuetype Kangaroo {}; > > #pragma ID Kangaroo "RMI:org.omg.customRMI.alpha.bravo.Kangaroo:87654321ABCDEF01:9876543210FEDCBA" > > }; > > }; > > }; > > }; > > }; > > > > 4. In section 1.4.10, change paragraph 1a to read: > > "a. octet - Format version. 1 or 2." > > 5. In the same section, change paragraph 1d to read: > > "d. (optional) Additional data written by writeObject, encoded as specified below. For format version 1, the data is written "as is". For format version 2, the data is enclosed within a CDR custom valuetype with no codebase and repid > > "RMI:org.omg.custom." where is the fully-qualified name of the class whose writeObject method is being invoked." > > 6. In the same section, at the end of subsection 1, add the following: > > "For interoperability using GIOP 1.2, format version 1 must be sent, except where the RMICustomStreamFormat service context (see section 1.4.11) has previously been received on the same connection. The same format version rules apply to RMI custom > > data sent within an encapsulation as part of the UnknownExceptionInfo service context. This is the only case where RMI custom data can appear as part of an encapsulation." > > 7. Add a new section 1.4.11 "RMICustomStreamFormat Service Context", renumbering the existing section 1.4.11 and subsequent sections: > > "The RMICustomStreamFormat service context contains a CDR encapsulation of the RMICustom::StreamFormat type, defined as follows. > > > > module RMICustom { > > > > struct StreamFormat { > > octet maxSerializableFormatVersion; // highest serializable format supported > > octet minSerializableFormatVersion; // lowest serializable format supported > > octet maxExternalizableFormatVersion; // highest externalizable format supported > > octet minExternalizableFormatVersion; // lowest externalizable format supported > > }; > > > > }; > > > > This service context allows ORBs to indicate that they are able to accept a higher or lower stream format version (or both) for RMI custom data than the format versions required by the GIOP level of the message used to send the valuetype containing > > RMI custom data. > > > > The RMICustomStreamFormat service context can be sent on any GIOP message that can carry a service context. An ORB that receives this service context is allowed to send on the same connection RMI custom data with any stream format version that is > > within the bounds specified by the service context. It is always permissible to send any stream format version whose support is defined as mandatory for the message GIOP level being sent." > > 8. Add new section 1.5.1.3 "ValueOutputStream" and 1.5.1.4 "ValueInputStream", renumbering the existing section 1.5.1.3 and subsequent sections: > > "The interface org.omg.CORBA.portable.ValueOutputStream defines methods that allow serialization of custom-marshaled RMI/IDL objects to GIOP streams. > > > > // Java > > package org.omg.CORBA.portable; > > public interface ValueOutputStream { > > void start_value(java.lang.String rep_id); > > void end_value(); > > } > > > > The interface org.omg.CORBA.portable.ValueInputStream defines methods that allow deserialization of custom-marshaled RMI/IDL objects from GIOP streams. > > > > // Java > > package org.omg.CORBA.portable; > > public interface ValueInputStream { > > void start_value(); > > void end_value(); > > } > > > > " > > 9. In existing section 1.5.1.3, add a new paragraph at the end of the "Execution Model for Serialization" section as follows: > > "If the ORB supports custom serialization format 2 as described in item 1a of section 1.4.10, then the ORB streams passed to the writeValue and readValue methods of the ValueHandler object must implement the ValueOutputStream and ValueInputStream > > interfaces respectively. For output streams, the start_value() method ends any currently open chunk, writes a valuetype header for a nested custom valuetype (with a null codebase and the specified repository ID), and increments the valuetype > > nesting depth. The end_value() method ends any currently open chunk, writes the end tag for the nested custom valuetype, and decrements the valuetype nesting depth. For input streams, the start_value() method reads a valuetype header for a nested > > custom valuetype and increments the valuetype nesting depth. The end_value() method reads the end tag for the nested custom valuetype (after skipping any data that precedes the end tag) and decrements the valuetype nesting depth." > > 10. In existing section 1.5.1.3, add a new paragraph at the end of the "Value Marshaling" section as follows: > > "If the ValueHandler supports custom serialization format 2 as described in item 1a of section 1.4.10, then the ValueHandler object must call the ValueOutputStream.start_value() and ValueOutputStream.end_value() methods of the ORB stream before and > > after writing the data specified by item 1d of section 1.4.10. The rep_id string passed to the start_value() method must be "RMI:org.omg.custom.::" where is the fully-qualified name of the class whose writeObject method is being invoked and and are > > the class's hashcode and SUID." > > 11. In existing section 1.5.1.3, add a new paragraph at the end of the "Value Unmarshaling" section as follows: > > "If the ValueHandler supports custom serialization format 2 as described in item 1a of section 1.4.10, then the ValueHandler object must call the ValueInputStream.start_value() and ValueInputStream.end_value() methods of the ORB stream before and > > after reading the data specified by item 1d of section 1.4.10." > > > > The following changes are proposed in the CORBA specification, document number formal/2001-09-34: > > > > 1. In section 13.7.1, add the following ServiceId constant definition: > > > > const ServiceId RMICustomStreamFormat = 15; > > > > and the following bullet: > > o RMICustomStreamFormat identifies a CDR encapsulation of the RMICustom::StreamFormat defined in the Java to IDL mapping specification (formal/xx-xx-xx), section 1.4.11. > > 2. In section 13.9, add a row to the table for feature "RMICustomStreamFormat service context" showing that it is optional for version 1.2. > > 3. In section A.2, add a row to the table for ServiceId tag value "RMICustomStreamFormat = 15" defined in the Java to IDL mapping specification (formal/xx-xx-xx), section 1.4.11. > > > > Actions taken: > > December 20, 1999: received issue > > > > Issue 4591: Behavior of writeByte, writeChar, writeBytes, writeChars > > > > Click here for this issue's archive. > > Source: Sun Microsystems (Mr. Everett Anderson, mailto:%20everett.anderson@sun.com) > > Nature: Uncategorized Issue > > Severity: > > Summary: A Serializable or Externalizable can define methods which take java.io.ObjectOutputStreams. There are four methods on that class that are well defined in Java, but the mapping to CORBA may need to be clarified: writeByte(int) writeChar(int) > > writeBytes(String) writeChars(String) Please see the Java docs for these methods: http://java.sun.com/j2se/1.3/docs/api/java/io/DataOutput.html#writeByte(int) http://java.sun.com/j2se/1.3/docs/api/java/io/DataOutput.html#writeChar(int) > > http://java.sun.com/j2se/1.3/docs/api/java/io/DataOutput.html#writeBytes(java.lang.String) http://java.sun.com/j2se/1.3/docs/api/java/io/DataOutput.html#writeChars(java.lang.String) Based on those detailed definitions, I'd say the mapping would be: > > writeByte(int) * write_octet of the lower 8 bits of the int writeChar(int) * two write_octet calls in the order as shown in the java docs writeBytes(String) * For each char in the String, call charAt, take the lower 8 bits, and call write_octet > > writeChars(String) * For each char in the String, call charAt, split the char into two bytes, and make two write_octet calls in the order shown in the java docs Another interpretation might be to use wstrings or wchar arrays for writeChars. The problem > > I see there is that since there isn't a readChars method, the user will be using the DataInput methods readFully or multiple readChar calls to reconstruct the String. Since the wire format of wstrings/wchars depends on the code set, he might start to > > see code set specific bytes instead of the expected bytes detailed in the java docs of writeChars and writeChar. > > Proposed Resolution: > > As proposed in the issue summary, the stream data written by these APIs for RMI-IIOP needs to be consistent with Java serialization, since otherwise some classes that serialize and deserialize correctly using Java serialization or RMI-JRMP would not > > serialize and deserialize correctly using RMI-IIOP. > > > > However, care needs to be taken when making this change to avoid breaking existing applications. This is because > > > > 1. the revised specification for writeBytes(String) is incompatible with the way this method is currently implemented by certain JDK ORBs (specifically IBM's J2SE 1.3 and Sun's J2SE 1.3), and > > 2. the change may affect application code as well as the ORB. > > > > The incompatibility is that the writeBytes() method currently converts characters to bytes using the platform's default character encoding, but the new specification doesn't do this. Therefore, any platform whose default character encoding is not > > ISO8859-1 (e.g., IBM zOS) will send different data for writeBytes() with the revised specification, and any application-defined Java readObject() methods that consume this data (on any platform) will have to be modified to consume unconverted data > > instead of converted data. Similarly, IDL custom valuetype unmarshal method implementations that consume data written by writeBytes() may have to be modified in the same way. > > > > Following established principles for incompatible changes to the J2SE platform, this change should only be made with adequate warning and on a major release boundary of the J2SE platform (e.g, 1.4 or 1.5) rather than on a minor release boundary (e.g., > > 1.4.x) or a service update (e.g., 1.4.0_xx). > > Proposed Revised Text: > > In the second last paragraph of section 1.4.10, add the following text before the last sentence of the paragraph: > > "Java ints and strings written by the writeByte, writeChar, writeBytes, and writeChars methods of java.io.ObjectOutputStream are marshaled as specified by the definitions of these methods in the java.io.DataOutput interface." > > Actions taken: > > October 4, 2001: received issue > > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > > > -- > Simon C Nash, Chief Technical Officer, IBM Java Technology > Tel. +44-1962-815156 Fax +44-1962-818999 Hursley, England > Internet: nash@hursley.ibm.com Lotus Notes: Simon Nash@ibmgb > -- Jeff Mischkinsky jmischki@dcn.davis.ca.us +1 530-758-9850 jeff.mischkinsky@oracle.com +1 650-506-1975 s more B-data to be written, while everything else stays the > > same. When the old B code unmarshals a new B instance, it will not fully consume all the B-data and this will cause the unmarshalling of the C-data to be incorrect since the stream will be out of sync. > > > > The problem is caused by a deficiency in the stream encoding for custom serialized data. In the example above, the existing format does not identify the boundary between the B-data and the C-data, so it is impossible for the unmarshalling code in the > > ValueHandler to skip over any extraneous B-data before it starts to read the C-data. > > > > The proposed resolution is to enclose the B-data in a nested valuetype. The repid for the nested valuetype must be specified by the Java to IDL mapping so that it can be read by IDL custom unmarshalling code in other languages. The repid is > > "RMI:org.omg.custom.::" where is the fully-qualified name of the class whose writeObject method is being invoked and and are the class's hashcode and SUID. The other language implementation has to provide a suitable custom valuetype implementation for > > the nested valuetype and may have to provide a value factory that maps the standard repid to this implementation. For Java unmarshalling, the ValueHandler will understand the special repid and process its contents as RMI-IIOP custom serialized data. > > > > Since valuetype headers and end tags are written by the ORB's OutputStream and not by the ValueHandler, it will be necessary to add new APIs to the OutputStream for the ValueHandler to call when marshalling custom valuetype data. These new APIs will > > write the valuetype header and end tag for the nested valuetype, keep track of the valuetype nesting level, and take care of chunking for the nested valuetype. Rather than adding yet another level of OutputStream subclassing, the new APIs are defined > > within an interface called org.omg.CORBA.portable.ValueOutputStream which will be implemented by the output stream passed to the ValueHandler. The new interface is defined as follows: > > > > public interface ValueOutputStream { > > void start_value(java.lang.String rep_id); > > void end_value(); > > } > > > > When the ValueHandler encounters a class with custom serialized data, it calls the start_value() method on the output stream that it was given, passing the special custom repid for the custom-serialized class. The output stream ends the currently open > > chunk, writes a valuetype header for the nested custom valuetype, and increments the valuetype nesting depth. It then returns to the ValueHandler, which calls writeObject() to write the custom serialized data. When writeObject() returns, the > > ValueHandler calls end_value() on the stream to end the current chunk, write the end tag, and decrement the valuetype nesting depth. > > > > When unmarshalling a class with custom serialized data, the ValueHandler needs to call the ORB's input stream to consume the valuetype header and end tag. A new interface org.omg.CORBA.portable.ValueInputStream that defines the necessary APIs will be > > implemented by the input stream. This interface is defined as follows: > > > > public interface ValueInputStream { > > void start_value(); > > void end_value(); > > } > > > > The ValueHandler calls the start_value() method on the input stream. The input stream reads the nested valuetype header and increments the valuetype nesting depth, then returns to the ValueHandler. The ValueHandler reads the custom serialized data, then > > calls end_value() which consumes the end tag (after skipping any data that precedes the end tag) and decrements the valuetype nesting depth. > > > > For languages other than Java, the nested valuetype must be handled explicitly by calling write_Value() and read_Value() within the implementations of the marshal() and unmarshal() methods. To assist in the implementation of these methods, a new IDL > > valuetype is generated by the Java to IDL mapping. In the example above, this would be: > > > > module org { > > module omg { > > module customRMI { > > custom valuetype B {}; > > #pragma ID B "RMI:org.omg.customRMI.B:1234567890ABCDEF:FEDCBA0987654321" > > }; > > }; > > }; > > > > This would be implemented in C++ (for example) with marshal() and unmarshal() methods that read and write the appropriate data. > > > > The new stream format would be identified by a format version of 2 (currently 1). Because GIOP 1.2 uses format version 1, support for format version 2 cannot be made mandatory until GIOP 1.3. In order to permit ORBs to use stream format 2 as an option > > over GIOP 1.2, a new service context is defined as follows: > > > > module RMICustom { > > > > struct StreamFormat { > > octet maxSerializableFormatVersion; // highest serializable format supported > > octet minSerializableFormatVersion; // lowest serializable format supported > > octet maxExternalizableFormatVersion; // highest externalizable format supported > > octet minExternalizableFormatVersion; // lowest externalizable format supported > > }; > > > > }; > > > > together with a new ServiceId constant: > > > > const ServiceId RMICustomStreamFormat = 15; > > > > RMICustomStreamFormat identifies a CDR encapsulation of the RMICustom::StreamFormat defined above. It allows ORBs to indicate that they are able to accept a higher or lower stream format version (or both) for RMI custom data than the format required by > > the GIOP level of the message used to send the valuetype containing RMI custom data. > > > > The RMICustomStreamFormat service context can be sent on any GIOP message that can carry a service context. An ORB that receives this service context is allowed to send on the same connection RMI custom data with any stream format version that is within > > the bounds specified by the service context. It is always permissible to send any stream format version whose support is defined as mandatory for the message GIOP level being sent. > > > > RMI custom data sent within an encapsulation as part of the UnknownExceptionInfo service context must have a stream format version that is compatible with either > > > > * an RMICustomStreamFormat service context previously received over the same connection, if such a service context has been received on this connection, or > > * any stream format version whose support is mandatory for the GIOP level of the GIOP message carrying the UnknownExceptionInfo service context. > > > > This is the only case where RMI custom data can appear as part of an encapsulation. > > Proposed Revised Text: > > The following changes are proposed in the Java to IDL mapping specification, document number ptc/2001-06-15: > > > > 1. In section 1.3.5.6, add the following at the end of the second paragraph: > > "An additional IDL custom valuetype in the module ::org::omg::customRMI is also generated to assist with marshaling and unmarshaling instances of the class. See section 1.3.5.8 for details." > > 2. Add a new section 1.3.5.8 "Secondary custom valuetype", renumbering the existing section 1.3.5.8 and subsequent sections: > > "In addition to the primary mapping described above, an RMI/IDL value type containing a writeObject method is mapped to a secondary IDL custom valuetype. The module name for this valuetype is formed by taking the ::org::omg::customRMI prefix and > > then adding the primary mapped type Date: Thu, 10 Jan 2002 16:52:01 -0800 From: Vijaykumar Natarajan Subject: RE: Java to IDL 2001 RTF Vote 3 reminder In-reply-to: <3C3C975D.9CACE76A@hursley.ibm.com> To: "'Simon Nash'" Cc: java2idl-rtf@omg.org Reply-to: vnatarajan@borland.com Message-id: <006901c19a3a$2e3db060$c35a14ac@VIJAY1> Organization: Borland Software Corporation MIME-version: 1.0 X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 X-Mailer: Microsoft Outlook, Build 10.0.3311 Content-transfer-encoding: 7BIT Importance: Normal X-Priority: 3 (Normal) X-MSMail-priority: Normal Content-Type: text/plain; charset=us-ascii X-UIDL: !SXd9^O@e9;g#e9%nR!! Status: RO Simon, I would like to change Borland's vote for Vote 3, issue 3151 from abstain to NO. I believe this is way too much machinery to solve a corner case. Thanks, vijay > -----Original Message----- > From: Simon Nash [mailto:nash@hursley.ibm.com] > Sent: Wednesday, January 09, 2002 11:18 AM > To: Jishnu Mukerji; Harold Carr; Jeff Mischkinsky; Yoshitaka > Honishi; Andy Piper > Cc: java2idl-rtf@omg.org > Subject: Java to IDL 2001 RTF Vote 3 reminder > > > Just a reminder that vote 3 closes at 6pm US PST tommorrow, > January 10. The vote is available at > http://cgi.omg.org/pub/java2idl-rtf/Java_to_IDL_2001_RTF_Vote3.htm > Thanks. > > Simon > -- > Simon C Nash, Chief Technical Officer, IBM Java Technology > Tel. +44-1962-815156 Fax +44-1962-818999 Hursley, England > Internet: nash@hursley.ibm.com Lotus Notes: Simon Nash@ibmgb > Date: Thu, 10 Jan 2002 16:59:21 -0800 From: Harold Carr X-Mailer: Mozilla 4.51 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: Andy Piper CC: Simon Nash , Jishnu Mukerji , Vijay Natarajan , Yoshitaka Honishi , Xudong Chen , java2idl-rtf@omg.org Subject: Re: Java to IDL 2001 RTF Vote 3 References: <4.3.2.7.2.20020110123419.00b535e0@san-francisco.beasys.com> <4.3.2.7.2.20020110144607.00b16aa8@san-francisco.beasys.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii X-UIDL: )D*!!*Icd9`ead9Ne@!! Status: RO Earlier SUN voted: YES: 4591: Behavior of writeByte, writeChar, writeBytes, writeChars ABSTAIN: 3151: Stream format problem for custom marshalled RMI types In abstaining I mentioned that it seems like the vote on 3151 could not even be valid, since there is talk about changing the text of the resolution. I still think that is the case. And, just in case there is disagreement as to 3151 vote validity, SUN changes it vote: YES: 4591 NO: 3151 In other words, we need to keep 3151 open for more discussion. Cheers, H Date: Thu, 10 Jan 2002 15:55:54 -0800 From: Everett Anderson X-Mailer: Mozilla 4.73 [en] (Windows NT 5.0; U) X-Accept-Language: en,pdf,ja MIME-Version: 1.0 To: Simon Nash CC: Andy Piper , Jishnu Mukerji , Vijay Natarajan , Harold Carr , Yoshitaka Honishi , Xudong Chen , java2idl-rtf@omg.org, interop@omg.org, michi.henning@iona.com Subject: Re: Issue 4795 resolution options References: <4.3.2.7.2.20020109110723.0390c460@san-francisco.beasys.com> <4.3.2.7.2.20020110121949.00b28178@san-francisco.beasys.com> <3C3E0D17.A46C90F7@hursley.ibm.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii X-UIDL: 9O@!!eX > Andy, > > Andy Piper wrote: > > > > At 01:21 PM 1/10/2002 +0000, Simon Nash wrote: > > >Thanks for this response. 4795 is different from 3151 regarding versioning > > >of the streams. With 3151, the sender needs to know the capability of the > > >receiver to avoid sending it a format that it can't handle. If it detects > > >a down-level receiver, it can send a down-level form of the data that the > > >down-level receiver is able to handle. > > > > > >With 4795, even if a J2SE 1.4 sender knows it is talking to a legacy J2SE 1.3 > > >receiver, it cannot marshal J2SE 1.4 exceptions in a form that the legacy > > >J2SE 1.3 ORB can unmarshal. So knowing the receiver's capability (whether by > > >stream versioning or GIOP level) doesn't help here. The only way to make > > >J2SE 1.4 and J2SE 1.3 interoperate is for J2SE 1.4 to send the new format > > >(whichever one we decide to adopt) and for legacy J2SE 1.3 ORBs to apply a > > >service update to allow them to accept the new format. > > > > > >If we go to a new GIOP level with full inline valuetype information to solve > > >this, as you suggest, the need for a service update to existing products > > >still applies. However, this would allow the current service context > > >definition for UnknownExceptionInfo to remain unchanged instead of introducing > > >a new flavor of this that holds an any. > > > > > >So let me sketch out an option 4 to add to the list: > > > a. In GIOP 1.3, change the valuetype CDR representation to allow an optional > > > TypeCode to be included. > > > b. Require that exception valuetypes in UnknownExceptionInfo service > > > contexts > > > be encoded with TypeCode information when using GIOP 1.3. > > > > > >Would you prefer this to the current option 3? > > > > I guess. Yes. > > > OK, I'll write up a proposal for this. It would have been nice to get more > feedback, but we need some kind of straw man that we can vote on. Also, this > requires a change in the interop RTF to change the CDR format for valuetypes. > I think the best way to proceed is for the Java to IDL RTF to vote on an > option 4 proposal subject to the interop RTF making this change, then if the > Java to IDL vote passes I can ask Michi to initiate a fast-path vote in the > interop RTF. I'm copying Michi and the interop list on this reply to see if > there are any howls of protest about this. I'm also copying Everett, since > he might have some views on this. > > > It still grieves me that we are contemplating such complexity for unknown > > exceptions. Relying on a service update seems problematic to me. I would > > almost prefer that if the sender knows the receiver is going to choke that > > it marshals an UNKNOWN with UnknownExceptionInfo containing a null > > valuetype. At least then the sender could marshal something that it knows > > will be understood. > > > Even this might not work. I don't think it's possible to throw a null > exception. The only way to avoid a service update for J2SE 1.3 ORBs would be > a) tie the fix to GIOP 1.3 > b) when 1.4 ORBs marshal UnknownExceptionInfo over GIOP 1.2, they must > exactly emulate the J2SE 1.3 exception SUID/hashcode/data. > > a) is easy, but b) is very difficult. We could add machinery to do this, but > I fear the cure is worse than the disease. > > Simon > -- > Simon C Nash, Chief Technical Officer, IBM Java Technology > Tel. +44-1962-815156 Fax +44-1962-818999 Hursley, England > Internet: nash@hursley.ibm.com Lotus Notes: Simon Nash@ibmgb X-Sender: andyp@san-francisco.beasys.com X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Date: Fri, 11 Jan 2002 00:42:38 -0800 To: Harold Carr , Simon Nash From: Andy Piper Subject: Re: Java to IDL 2001 RTF Vote 3 Cc: Jishnu Mukerji , Vijay Natarajan , Jeff Mischkinsky , Yoshitaka Honishi , Xudong Chen , java2idl-rtf@omg.org, juergen@omg.org In-Reply-To: <3C3E21D2.19F455BA@sun.com> References: <3C34DF7A.32E0A64E@hursley.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-UIDL: _CMe9ZVV!!=-P!!(K>e9 I think that Jeff put it best. Its not that I am against it, its more that we are throwing in significant complexity late in the day and I am not convinced that it will achieve the overall goal. I mostly say this because I remain to be convinced that interop at this level with J2SE 1.3 will *ever* be possible, but also because we appear to be contemplating multiple, possibly overlapping versioning solutions. Given that current versioning technology (chunking and SendingContextRunTime) are both broken in J2SE 1.3 and one is broken in J2SE 1.4 I don't see the need for rushing it. Incidentally, who owns the ValueHandler (is it this group or Sun)? Many of the current proposals impact the ValueHandler and I'm coming to the conclusion that the ValueHandler is currently too tightly tied to the Orb infrastructure. For instance how do we tell the ValueHandler about versioning information? andy At 03:20 PM 1/10/2002 -0800, Harold Carr wrote: 1. Why is anyone against the versioning in the proposal? 2. Why just solve it going forward when versioning can solve it going backward as well? 3. Won't making the versioning dependent on GIOP 1.3 slow down the solution consideraby compared to including a versioning scheme in a Java-to-IDL solution? 4. This seems to be involving solutions to 4795 also, right? Date: Fri, 11 Jan 2002 16:15:32 +0000 From: Simon Nash Organization: IBM X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; I) X-Accept-Language: en MIME-Version: 1.0 To: Andy Piper CC: Harold Carr , Jishnu Mukerji , Vijay Natarajan , Jeff Mischkinsky , Yoshitaka Honishi , Xudong Chen , java2idl-rtf@omg.org, juergen@omg.org Subject: Re: Java to IDL 2001 RTF Vote 3 References: <3C34DF7A.32E0A64E@hursley.ibm.com> <4.3.2.7.2.20020111003136.00ba0ea8@san-francisco.beasys.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii X-UIDL: DV0!!J9P!!]pVd9'+3e9 Andy, The goal of the fix for issue 3151 is to fix a problem with valuetype versioning over RMI-IIOP. This problem does not affect interoperability between J2SE versions as far as I am aware. However, it could affect versioning of user-defined classes that are not part of the J2SE base, and it could affect J2SE classes in the future (e.g., it could affect interoperability between J2SE 1.4 and J2SE 1.5). I'm not sure what you mean by chunking being broken in J2SE 1.3. However, we (the OMG) are not responsible for implementation bugs. We are responsible for producing specifications that are correct and implementable. The current RMI-IIOP stream format does not correctly support Java serialization semantics. This is a serious problem, and we are lucky that it has not yet blown up as a critical situation. We need to put a fix in place so that ORB implementers (including the Sun J2SE ORB team) can pick up this fix and get it into their future product releases. Since this fix is by its nature quite a significant change (and no-one has been able to produce a fully worked out proposal that is simpler), the forthcoming CORBA 3.0 and GIOP 1.3 versions provide a suitable point at which to make this change. If we could close this in time for GIOP 1.3, then I would hope that it could become part of J2SE 1.5. It may also be possible for other ORB vendors to provide it as part of GIOP 1.3 before the J2SE 1.5 FCS date. The ValueHandler APIs are owned by this group. The ValueHandler implementation is owned by Sun, and any ORB vendor can override the Sun implementation with their own implementation if they wish. The ORB should not assume that it is running with its own ValueHandler, and the ValueHandler implementation should not assume that it is running with its own ORB. Therfore, all necessary information that is passed between ORB and ValueHandler must by by means of standard APIs that are defined in the Java to IDL mapping specification. Currently, the ORB passes versioning information to the ValueHandler by passing an org.omg.SendingContext.RunTime parameter on the readValue() call. If we are going to resolve issue 4795 by allowing versioning information to be passed in the form of a TypeCode instead, then we will need a new overload for readValue() that takes a TypeCode parameter instead of a SendingContext.RunTime parameter. This doesn't affect issue 3151, though. Simon Andy Piper wrote: > > I think that Jeff put it best. Its not that I am against it, its more that > we are throwing in significant complexity late in the day and I am not > convinced that it will achieve the overall goal. I mostly say this because > I remain to be convinced that interop at this level with J2SE 1.3 will > *ever* be possible, but also because we appear to be contemplating > multiple, possibly overlapping versioning solutions. Given that current > versioning technology (chunking and SendingContextRunTime) are both broken > in J2SE 1.3 and one is broken in J2SE 1.4 I don't see the need for rushing it. > > Incidentally, who owns the ValueHandler (is it this group or Sun)? Many of > the current proposals impact the ValueHandler and I'm coming to the > conclusion that the ValueHandler is currently too tightly tied to the Orb > infrastructure. For instance how do we tell the ValueHandler about > versioning information? > > andy > > At 03:20 PM 1/10/2002 -0800, Harold Carr wrote: > >1. Why is anyone against the versioning in the proposal? > > > >2. Why just solve it going forward when versioning can solve it going > >backward as well? > > > >3. Won't making the versioning dependent on GIOP 1.3 slow down the > >solution consideraby compared to including a versioning scheme in a > >Java-to-IDL solution? > > > >4. This seems to be involving solutions to 4795 also, right? -- Simon C Nash, Chief Technical Officer, IBM Java Technology Tel. +44-1962-815156 Fax +44-1962-818999 Hursley, England Internet: nash@hursley.ibm.com Lotus Notes: Simon Nash@ibmgb Date: Fri, 11 Jan 2002 10:05:19 +0000 From: Simon Nash Organization: IBM X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; I) X-Accept-Language: en MIME-Version: 1.0 To: Jishnu Mukerji CC: java2idl-rtf@omg.org, everett.anderson@Sun.COM, andyp@bea.com, interop@omg.org Subject: Re: Issue 3151 and GIOP 1.3 References: <3C25F777.B7F351B1@hursley.ibm.com> <3C3E2197.FD2DA152@hp.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii X-UIDL: Z-D!!X > Simon Nash wrote: > > > > It is my understanding that GIOP 1.3 will be adopted soon and will differ > > from GIOP 1.2 only in minor ways, e.g., by adding one or two operations to > > CORBA::Object. If this is correct, then we could consider tying the fix > > for issue 3151 to GIOP 1.3 rather than adding a service context to allow > > the new stream format to be used with GIOP 1.2. This change would simplify > > the proposal. > > > > I'd appreciate any comments on this, as I'm in the process of writing up a > > proposed resolution for voting. Thanks. > > Looking back at the history of GIOP 1.3 discussion, it seems to me that > we do need to delay GIOP 1.3 at least until the Components and Firewall > FTF complete their tasks. Given that Firewall is completing final > submissions at this meeting, I don't see how this can be achieved before > the middle of the year at the earliest (Orlando meeting?) We should then > take at least one meeting cycle of serious work in the Interop RTF to > figure out how to throw these disperate pieces of changes to GIOP coming > in from various directions need to be aligned together, just for > everyones sanity's sake. > > Here are the significant components going into various GIOP changes: > > 1. Components spec (which is ready now) adds _component (or some such). > They did punt on the other thing which should probably be resurrected in > the Interop RTF at some point. Anyway either one of those requires a > minor number rev. > > 2. Firewall spec, adds random significant things to GIOP. This will > probably not be finalized until say 3Q02. > > 3. A few proposals that are floating around in Core, Java-IDL etc RTFs > add bits and pieces to GIOP that should require revving minor number. > Some of these could be ready late 1Q/early 2Q. > > 4. Some others that I may have missed (can anyone think of anything > else? > > Now, here is the rub. If we delay GIOP 1.3 all the way to 3Q, that means > that in effect interoperable Components specification will not be > available until 3Q02, which may not be acceptable to those that plan to > use the Components spec. Also changes that happen as late as 3/4Q 02 > will also miss the next J2EE bus, as far as I can tell. So we need to > come up with a plan that minimizes the pain for everyone and maximizes > the returns. This may require two minor revisions of GIOP suitably timed > through 2002. But this requires some serious headbashing and I propose > that we do some serious brainstorming on this matter and come up with a > plan by the end of the Anaheim meeting and follow through oin it with > due diligence. We need to decide: > > i. How many minor revs and what timeframes. > > ii. Contents of each minor rev - perhaps do a triage of the infamous > wishlist (actually more like a lets punt list) and see what items from > it can be realisitically handled in the given timeframes. > > iii. Keep in mind that it is inappropriate to delay already adopted > specs from being formally published based on speculative contents of > speculative minor revisions.:-) > > What says all of you? > > Cheers, > > Jishnu. -- Simon C Nash, Chief Technical Officer, IBM Java Technology Tel. +44-1962-815156 Fax +44-1962-818999 Hursley, England Internet: nash@hursley.ibm.com Lotus Notes: Simon Nash@ibmgb Date: Fri, 11 Jan 2002 09:53:24 +0000 From: Simon Nash Organization: IBM X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; I) X-Accept-Language: en MIME-Version: 1.0 To: Jishnu Mukerji , Vijay Natarajan , Harold Carr , Jeffrey Mischkinsky , Yoshitaka Honishi , Andy Piper , Xudong Chen CC: java2idl-rtf@omg.org, everett.anderson@Sun.COM Subject: Revised proposal for issue 3151 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=iso-8859-1 X-UIDL: bYP!!)P2e9Y^%!!Mj7!! The proposal for issue 3151 failed to pass in vote 3. The reasons stated included concern about the stream versioning mechanism and the complexity of the proposal. I have therefore revised the proposal to remove the stream versioning, which also significantly reduces the complexity. I would like to put this revised proposal up for voting in vote 4, which I plan to send out on Monday 17th. If anyone has comments, questions, or suggested changes, please let me know. The only change from the previous proposal is removal of stream versioning. Here's an example of a case that doesn't work: public class A implements java.io.Serializable { // any data } public class B extends A { // any data private void writeObject(java.io.ObjectOutputStream stream) { // any code } private void readObject(java.io.ObjectInputStream stream) { // any code } } public class C extends B { // any data } At present, an instance of C is marshalled as C-header A-data B-data C-data C-endtag where all the data is written and read by the ValueHandler and is chunked according to the valuetype chunking rules. Suppose some change to the writeObject() implementation in class B cause secondary valuetype is the same as the name of the primary IDL custom value type to which the RMI/IDL value type was mapped. The secondary valuetype has no inheritance, data members, methods, or initializers. It has a #pragma ID specifying a repository ID formed by taking the repository ID of the primary custom valuetype and prefixing the Java package name with "org.omg.customRMI.". The secondary custom valuetype represents the enclosure of writeObject data that is wriitten to the serialization stream when the primary custom valuetype or any of its subclasses is serialized using format version 2, as described in item 1d of section 1.4.10. For IDL custom marshaling and unmarshaling of the primary mapped IDL valuetype, the marshal() and unmarshal() methods can call write_Value() and read_Value() to write and read the nested valuetype enclosure. This will cause the marshal() and unmarshal() methods of the secondary mapped IDL valuetype to be called to write and read the custom serialized data. 3. In existing section 1.3.5.9, add the following to the generated IDL: module org { module omg { module customRMI { module alpha { module bravo { custom valuetype Kangaroo {}; #pragma ID Kangaroo "RMI:org.omg.customRMI.alpha.bravo.Kangaroo:87654321ABCDEF01:9876543210FEDCBA" }; }; }; }; }; 4. In section 1.4.10, change paragraph 1a to read: a. octet - Format version. 1 or 2. 5. In the same section, change paragraph 1d to read: d. (optional) Additional data written by writeObject, encoded as specified below. For format version 1, the data is written "as is". For format version 2, the data is enclosed within a CDR custom valuetype with no codebase and repid "RMI:org.omg.custom." where is the fully-qualified name of the class whose writeObject method is being invoked. 6. In the same section, at the end of subsection 1, add the following: For RMI/IDL value types marshaled within GIOP 1.2 messages (including the UnknownExceptionInfo service context), format version 1 must be used. For RMI/IDL value types marshaled within GIOP 1.3 messages (including the UnknownExceptionInfo service context), format version 2 must be used. 7. Add new section 1.5.1.3 "ValueOutputStream" and 1.5.1.4 "ValueInputStream", renumbering the existing section 1.5.1.3 and subsequent sections: The interface org.omg.CORBA.portable.ValueOutputStream defines methods that allow serialization of custom-marshaled RMI/IDL objects to GIOP streams. // Java package org.omg.CORBA.portable; public interface ValueOutputStream { void start_value(java.lang.String rep_id); void end_value(); } The interface org.omg.CORBA.portable.ValueInputStream defines methods that allow deserialization of custom-marshaled RMI/IDL objects from GIOP streams. // Java package org.omg.CORBA.portable; public interface ValueInputStream { void start_value(); void end_value(); } 8. In existing section 1.5.1.3, add a new paragraph at the end of the "Execution Model for Serialization" section as follows: The ORB streams passed to the writeValue and readValue methods of the ValueHandler object must implement the ValueOutputStream and ValueInputStream interfaces respectively. For output streams, the start_value() method ends any currently open chunk, writes a valuetype header for a nested custom valuetype (with a null codebase and the specified repository ID), and increments the valuetype nesting depth. The end_value() method ends any currently open chunk, writes the end tag for the nested custom valuetype, and decrements the valuetype nesting depth. For input streams, the start_value() method reads a valuetype header for a nested custom valuetype and increments the valuetype nesting depth. The end_value() method reads the end tag for the nested custom valuetype (after skipping any data that precedes the end tag) and decrements the valuetype nesting depth. 9. In existing section 1.5.1.3, add a new paragraph at the end of the "Value Marshaling" section as follows: If the ORB output stream passed to the ValueHandler supports the ValueOutputStream interface, then the ValueHandler object must call the ValueOutputStream.start_value() and ValueOutputStream.end_value() methods of the ORB stream before and after writing the data specified by item 1d of section 1.4.10. The rep_id string passed to the start_value() method must be "RMI:org.omg.custom.::" where is the fully-qualified name of the class whose writeObject method is being invoked and and are the class's hashcode and SUID. 10. In existing section 1.5.1.3, add a new paragraph at the end of the "Value Unmarshaling" section as follows: If the ORB input stream passed to the ValueHandler supports the ValueInputStream interface, then the ValueHandler object must call the ValueInputStream.start_value() and ValueInputStream.end_value() methods of the ORB stream before and after reading the data specified by item 1d of section 1.4.10. Simon -- Simon C Nash, Chief Technical Officer, IBM Java Technology Tel. +44-1962-815156 Fax +44-1962-818999 Hursley, England Internet: nash@hursley.ibm.com Lotus Notes: Simon Nash@ibmgb s more B-data to be written, while everything else stays the same. When the old B code unmarshals a new B instance, it will not fully consume all the B-data and this will cause the unmarshalling of the C-data to be incorrect since the stream will be out of sync. The problem is caused by a deficiency in the stream encoding for custom serialized data. In the example above, the existing format does not identify the boundary between the B-data and the C-data, so it is impossible for the unmarshalling code in the ValueHandler to skip over any extraneous B-data before it starts to read the C-data. Having carefully considered a number of possible solutions to this problem, I think Bob Scheifler's proposal of enclosing the B-data in a nested valuetype is the best approach. The repid for the nested valuetype must be specified by the Java to IDL mapping so that it can be read by IDL custom unmarshalling code in other languages. I propose that this repid be "RMI:org.omg.custom.::" where is the fully-qualified name of the class whose writeObject method is being invoked and and are the class's hashcode and SUID. The other language will have to provide a suitable custom valuetype implementation for the nested valuetype and may have to provide a value factory that maps the standard repid to this implementation. For Java unmarshalling, the ValueHandler will understand the special repid and process its contents as RMI-IIOP custom serialized data. Since valuetype headers and end tags are written by the ORB's OutputStream and not by the ValueHandler, it will be necessary to add new APIs to the OutputStream for the ValueHandler to call when marshalling custom valuetype data. These new APIs will write the valuetype header and end tag for the nested valuetype, keep track of the valuetype nesting level, and take care of chunking for the nested valuetype. Rather than adding yet another level of OutputStream subclassing, I propose adding the new APIs to an interface called org.omg.CORBA.portable.ValueOutputStream which will be implemented by the output stream passed to the ValueHandler. The new interface is defined as follows: public interface ValueOutputStream { void start_value(java.lang.String rep_id); void end_value(); } When the ValueHandler encounters a class with custom serialized data, it calls the start_value() method on the output stream that it was given, passing the special custom repid for the custom-serialized class. The output stream ends the currently open chunk, writes a valuetype header for the nested custom valuetype, and increments the valuetype nesting depth. It then returns to the ValueHandler, which calls writeObject() to write the custom serialized data. When writeObject() returns, the ValueHandler calls end_value() on the stream to end the current chunk, write the end tag, and decrement the valuetype nesting depth. When unmarshalling a class with custom serialized data, the ValueHandler needs to call the ORB's input stream to consume the valuetype header and end tag. A new interface org.omg.CORBA.portable.ValueInputStream that defines the necessary APIs will be implemented by the input stream. This interface is defined as follows: public interface ValueInputStream { void start_value(); void end_value(); } The ValueHandler calls the start_value() method on the input stream. The input stream reads the nested valuetype header and increments the valuetype nesting depth, then returns to the ValueHandler. The ValueHandler reads the custom serialized data, then calls end_value() which consumes the end tag (after skipping any data that precedes the end tag) and decrements the valuetype nesting depth. For languages other than Java, the nested valuetype must be handled explicitly by calling write_Value() and read_Value() within the implementations of the marshal() and unmarshal() methods. To assist in the implementation of these methods, a new IDL valuetype is generated by the Java to IDL mapping. In the example above, this would be: module org { module omg { module customRMI { custom valuetype B {}; #pragma ID B "RMI:org.omg.customRMI.B:1234567890ABCDEF:FEDCBA0987654321" }; }; }; This would be implemented in C++ (for example) with marshal() and unmarshal() methods that read and write the appropriate data. The new stream format would be identified by a format version of 2 (currently 1). RMI-IIOP custom valuetypes sent in GIOP 1.3 messages (including the UnknownExceptionInfo service context of a GIOP 1.3 reply) would require the use of format version 2. Here's a formal proposal for the changes needed to the Java to IDL mapping spec, document number ptc/2001-06-15: 1. In section 1.3.5.6, add the following at the end of the second paragraph: An additional IDL custom valuetype in the module ::org::omg::customRMI is also generated to assist with marshaling and unmarshaling instances of the class. See section 1.3.5.8 for details. 2. Add a new section 1.3.5.8 "Secondary custom valuetype", renumbering the existing section 1.3.5.8 and subsequent sections: In addition to the primary mapping described above, an RMI/IDL value type containing a writeObject method is mapped to a secondary IDL custom valuetype. The module name for this valuetype is formed by taking the ::org::omg::customRMI prefix and then adding the primary mapped type Date: Fri, 11 Jan 2002 18:22:18 -0500 From: Jishnu Mukerji Organization: Hewlett-Packard Company X-Mailer: Mozilla 4.78 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: Simon Nash Cc: java2idl-rtf@omg.org, everett.anderson@Sun.COM, andyp@bea.com, interop@omg.org Subject: Re: Issue 3151 and GIOP 1.3 References: <3C25F777.B7F351B1@hursley.ibm.com> <3C3E2197.FD2DA152@hp.com> <3C3EB8DF.92FA0550@hursley.ibm.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii X-UIDL: &(cd9L^g!!0Xad9ER:!! Status: RO Simon Nash wrote: > > Jishnu, > Your proposal sounds good to me. If the decision is to do two minor revs, > then I would want to see critical fixes in the first minor rev, and less > critical improvements in the second rev. Of course there will be lots of > debate about which items fit into which category :-) > > Simon My suspicion is that the first rev will consist of everything that has already been adopted, since no one really has a right to unilaterlly withold publication of things that have already been adopted. Then the second rev will be the most important things that can be handled by RTF. Then one might need a major rev for stuff that requires the issuance of an RFP. I suspect the fist rev will be between now and mid 2002, and the second one will be 3/4Q 2002, when among other things Firewall completes finalization. That is of course just my somewhat educated guess keeping overall fairness and sanity of the process forefront in ones mind. But we should of course discuss this whole thing carefully in Anaheim. I am preparing a draft of Chapters 13 and 15 with everything that has already completed adoption process added on to provide concrete grounds to base the discussion on. This of course does not include 4337, unless it happens to pass. Cheers, Jishnu. Date: Fri, 11 Jan 2002 23:30:21 +0000 From: Simon Nash Organization: IBM X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; I) X-Accept-Language: en MIME-Version: 1.0 To: Jishnu Mukerji CC: java2idl-rtf@omg.org, everett.anderson@Sun.COM, andyp@bea.com, interop@omg.org Subject: Re: Issue 3151 and GIOP 1.3 References: <3C25F777.B7F351B1@hursley.ibm.com> <3C3E2197.FD2DA152@hp.com> <3C3EB8DF.92FA0550@hursley.ibm.com> <3C3F73AA.F86BDF1F@hp.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii X-UIDL: U+=!!;W0!!L$?e9:+K!! Status: RO Jishnu, As far as I know, the only change that has been adopted so far is the _component message. I asked this question on the mailing list some time ago and no-one came up with anything else. So I await your complete list with great interest. I am not going to be in Anaheim, but I would like to take part in the discussion if possible. Could a speaker phone and dial-in number be arranged? I will even offer to provide the dial-in number. Simon Jishnu Mukerji wrote: > > Simon Nash wrote: > > > > Jishnu, > > Your proposal sounds good to me. If the decision is to do two minor revs, > > then I would want to see critical fixes in the first minor rev, and less > > critical improvements in the second rev. Of course there will be lots of > > debate about which items fit into which category :-) > > > > Simon > > My suspicion is that the first rev will consist of everything that has > already been adopted, since no one really has a right to unilaterlly > withold publication of things that have already been adopted. Then the > second rev will be the most important things that can be handled by RTF. > Then one might need a major rev for stuff that requires the issuance of > an RFP. I suspect the fist rev will be between now and mid 2002, and the > second one will be 3/4Q 2002, when among other things Firewall completes > finalization. That is of course just my somewhat educated guess keeping > overall fairness and sanity of the process forefront in ones mind. > > But we should of course discuss this whole thing carefully in Anaheim. I > am preparing a draft of Chapters 13 and 15 with everything that has > already completed adoption process added on to provide concrete grounds > to base the discussion on. This of course does not include 4337, unless > it happens to pass. > > Cheers, > > Jishnu. -- Simon C Nash, Chief Technical Officer, IBM Java Technology Tel. +44-1962-815156 Fax +44-1962-818999 Hursley, England Internet: nash@hursley.ibm.com Lotus Notes: Simon Nash@ibmgb Date: Sun, 13 Jan 2002 21:14:30 -0800 (PST) From: Everett Anderson X-Sender: eea1@wampum To: Simon Nash cc: Jishnu Mukerji , java2idl-rtf@omg.org, everett.anderson@Sun.COM, andyp@bea.com, interop@omg.org Subject: Re: Issue 3151 and GIOP 1.3 In-Reply-To: <3C3F758D.DE3D07C9@hursley.ibm.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-UIDL: VN!"!:ZBe9 Organization: Hewlett-Packard Company X-Mailer: Mozilla 4.78 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: Simon Nash Cc: java2idl-rtf@omg.org, everett.anderson@Sun.COM, andyp@bea.com, interop@omg.org Subject: Re: Issue 3151 and GIOP 1.3 References: <3C25F777.B7F351B1@hursley.ibm.com> <3C3E2197.FD2DA152@hp.com> <3C3EB8DF.92FA0550@hursley.ibm.com> <3C3F73AA.F86BDF1F@hp.com> <3C3F758D.DE3D07C9@hursley.ibm.com> Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii X-UIDL: o-1e9'4Sd9h`J!!X;8!! Simon Nash wrote: > > Jishnu, > As far as I know, the only change that has been adopted so far is the > _component message. I asked this question on the mailing list some time > ago and no-one came up with anything else. So I await your complete list > with great interest. It is the _component messsage and the new typecodes tk_component, tk_home and tk_event. Also, the typecodes tk_local and tk_abstract become mandatory in the new minor version. They are optional in 1.2 since there are versions of CORBA that run on 1.2 that do not know about these typecodes. We should have revved GIOP when these were added, to do it right, but unfortunately we did not. > I am not going to be in Anaheim, but I would like to take part in the > discussion if possible. Could a speaker phone and dial-in number be arranged? > I will even offer to provide the dial-in number. Mostprobably we will meet over lunch (if we do) one day and if that is the case it would be kind of hard to get a telephone in there, but we'll see what can be done. Jishnu. X-Sender: andyp@san-francisco.beasys.com X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Date: Mon, 14 Jan 2002 02:48:26 -0800 To: Simon Nash From: Andy Piper Subject: Re: Java to IDL 2001 RTF Vote 3 Cc: java2idl-rtf@omg.org, juergen@omg.org In-Reply-To: <3C3F0FA4.89DACC68@hursley.ibm.com> References: <3C34DF7A.32E0A64E@hursley.ibm.com> <4.3.2.7.2.20020111003136.00ba0ea8@san-francisco.beasys.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-UIDL: 0kKd9"U8e9-RK!!0Jj!! Hi Simon, At 04:15 PM 1/11/2002 +0000, Simon Nash wrote: The goal of the fix for issue 3151 is to fix a problem with valuetype versioning over RMI-IIOP. This problem does not affect interoperability Right, and I think it is a good solution for this. between J2SE versions as far as I am aware. However, it could affect versioning of user-defined classes that are not part of the J2SE base, and it could affect J2SE classes in the future (e.g., it could affect interoperability between J2SE 1.4 and J2SE 1.5). I think what I meant was that it was an incompatible change to the stream format. Generally in versioning I think you have these options: a) Lowest common denominator b) Receiver makes right c) Sender makes right d) Patch older versions Generally my order of preference would be a, b, c, d - but unfortunately the ease of implementation is usually d, c, b, a. In general the purpose of iiop and CORBA interop is for (a) to work. i.e. you define the format and everyone adheres to it. Unfortunately bugs in the spec and implementation make (a) almost impossible. (b) is next desirable because in general you do not have to know about all versions you instead know of someway to get what you want (e.g. skipping data in chunking). Unfortunately this is also difficult to achieve because it assumes that older versions were implemented correctly and usually they aren't (either because the spec was different then or because of bugs). (c) usually works but requires that each new version needs to know about all earlier versions, this can become somewhat problematic from a maintenance viewpoint. And (d) always works but tends to annoy customers. [There is of course (e) which is to upload the required change to the older version automatically which we have suggested to Sun.] (c) generally requires that you know the version of the stream. Which is what I believe we are considering in this instance. However, before we even consider (c) in this case I think we should at least make sure that (a) and (b) will be possible in the future. GIOP has at least achieved this tolerably well. You know you can always fallback to a lower rev. You know that you will be able to have at least some sort of conversation with your target. By tying the version change to a GIOP rev you can at least preserve these guarantees in a way that people and systems are used to. If we are going to step away from that kind of versioning I think we need to think very carefully about the implications and make sure we have covered all the bases. valuetypes are part of the CORBA spec. It seems to me that therefore the marshaled form of valuetypes be tied to the CORBA spec in some way and not be free-floating. Otherwise I think (a) will probably be unachievable. I'm not sure what you mean by chunking being broken in J2SE 1.3. However, The end-tag nesting level is calculated incorrectly. we (the OMG) are not responsible for implementation bugs. We are responsible for producing specifications that are correct and implementable. The current RMI-IIOP stream format does not correctly support Java serialization semantics. This is a serious problem, and we are lucky that it has not yet blown up as a critical situation. We need to put a fix in place so that ORB implementers (including the Sun J2SE ORB team) can pick up this fix and get it into their future product releases. I totally agree. As I said before I am not against the change, I am just questioning the additional work we are doing to try and make it work on older revs of the protocol. Since this fix is by its nature quite a significant change (and no-one has been able to produce a fully worked out proposal that is simpler), the forthcoming CORBA 3.0 and GIOP 1.3 versions provide a suitable point at which to make this change. If we could close this in time for GIOP 1.3, then I would hope that it could become part of J2SE 1.5. It may also be possible for other ORB vendors to provide it as part of GIOP 1.3 before the J2SE 1.5 FCS date. I also totally agree. The ValueHandler APIs are owned by this group. The ValueHandler implementation is owned by Sun, and any ORB vendor can override the Sun implementation with their own implementation if they wish. The ORB should not assume that it is running with its own ValueHandler, and the ValueHandler implementation should not assume that it is running with its own ORB. Therfore, all necessary information that is passed between ORB and ValueHandler must by by means of standard APIs that are defined in the Java to IDL mapping specification. Currently, the ORB passes versioning information to the ValueHandler by passing an org.omg.SendingContext.RunTime parameter on the readValue() call. If we are going to resolve issue 4795 by allowing versioning information to be passed in the form of a TypeCode instead, then we will need a new overload for readValue() that takes a TypeCode parameter instead of a SendingContext.RunTime Presumably its conceivable that this function could be used exclusively since RunTime essentially returns you a TypeCode anyway? I would actually welcome this since it pulls some protocol interactions out of the ValueHandler (i.e. contacting the foreign server). I'm now mulling over Everett's mail because he makes some very valid points... Thanks andy Date: Tue, 15 Jan 2002 22:42:02 +0000 From: Simon Nash Organization: IBM X-Mailer: Mozilla 4.72 [en] (Windows NT 5.0; I) X-Accept-Language: en MIME-Version: 1.0 To: java2idl-rtf@omg.org Subject: Updated proposal for issue 3151 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=iso-8859-1 X-UIDL: ;"od9o/S!!=/Q!!M" where is the fully-qualified name of the class whose writeObject method is being invoked. For externalizable objects: b. (optional) Data written by writeExternal, encoded as specified below. 5. In the same section, at the end of subsection 1, add the following: The default custom stream format is 1 for GIOP 1.2 and 2 for GIOP 1.3. For RMI/IDL custom value types marshaled within GIOP requests, a format version not greater than the default for the GIOP message level must be sent, except where the TAG_RMI_CUSTOM_MAX_STREAM_FORMAT tagged component (see section 1.4.11) is part of the IOR profile. For RMI/IDL custom value types marshaled within GIOP replies (including the UnknownExceptionInfo service context), a format version not greater than the default for the GIOP message level must be sent, except where the RMICustomMaxStreamFormat service context (see section 1.4.12) was sent on the associated GIOP request. 6. Add a new section 1.4.11 "TAG_RMI_CUSTOM_MAX_STREAM_FORMAT Component", renumbering the existing section 1.4.11 and subsequent sections: Although the IIOP level of an IOR specifies a default maximum stream format version for RMI/IDL custom value types marshaled as part of GIOP requests to this IOR, there are cases when it may be necessary to override this default. The TAG_RMI_CUSTOM_MAX_STREAM_FORMAT component has an associated value of type octet, encoded as a CDR encapsulation, designating the maximum stream format version for RMI/IDL custom value types that can be used in GIOP messages sent to this IOR. The TAG_RMI_CUSTOM_MAX_STREAM_FORMAT component can appear at most once in any IOR profile. For profiles supporting IIOP 1.2 or greater, it is optionally present. If this component is omitted, then the default maximum stream format version for RMI/IDL custom value types sent to this IOR is 1 for IIOP 1.2 and 2 for IIOP 1.3. 7. Add a new section 1.4.12 "RMICustomMaxStreamFormat Service Context", renumbering the existing section 1.4.11 and subsequent sections: Although the GIOP level of a request specifies a default maximum stream format version for RMI/IDL custom value types marshaled as part of the associated reply, there are cases when it may be necessary to override this default. RMICustomMaxStreamFormat identifies a CDR encapsulation of a single octet that specifies the highest RMI/IDL custom stream format version that can be used for RMI/IDL custom valuetypes marshaled within a GIOP reply associated with the GIOP request that carries this service context. If this service context is omitted from a GIOP request, then the default maximum stream format version for RMI/IDL custom value types marshaled within a GIOP reply associated with this request is 1 for GIOP 1.2 and 2 for GIOP 1.3. 8. Change the title of existing section 1.5.1.3 to "ValueHandler and ValueHandlerMultiFormat". Also change the first paragraph to: The interfaces javax.rmi.CORBA.ValueHandler and javax.rmi.CORBA.ValueHandlerMultiFormat define methods that allow serialization of Java objects to and from GIOP streams. Also add the following interface: //Java public interface ValueHandlerMultiFormat extends ValueHandler { byte getMaximumStreamFormatVersion(); void writeValue(org.omg.CORBA.portable.OutputStream out, java.io.Serializable value, byte streamFormatVersion); } and add the following paragraphs to this section before the "Execution Model for Serialization" subsection: The ValueHandlerMultiFormat interface introduces a method getMaximumStreamFormatVersion that returns the maximum stream format version for RMI/IDL custom value types that is supported by this ValueHandler object. The ValueHandler object must support the returned stream format version and all lower versions. The format versions currently defined are 1 and 2. See section 1.4.10 "Custom Marshaling Format" for more details. The ValueHandlerMultiFormat interface introduces an overloaded writeValue method that allows the ORB to pass the required stream format version for RMI/IDL custom value types. If the ORB calls this method, it must pass a stream format version between 1 and the value returned by the getMaximumStreamFormatVersion method inclusive, or else a BAD_PARAM exception with standard minor code (note to editor: number to be assigned by OMG) must be thrown. If the ORB calls the ValueHandler.writeValue method, stream format version 1 is implied. 9. Add new section 1.5.1.3 "ValueOutputStream" and 1.5.1.4 "ValueInputStream", renumbering the existing section 1.5.1.3 and subsequent sections: The interface org.omg.CORBA.portable.ValueOutputStream defines methods that allow serialization of custom-marshaled RMI/IDL objects to GIOP streams. // Java package org.omg.CORBA.portable; public interface ValueOutputStream { void start_value(java.lang.String rep_id); void end_value(); } The interface org.omg.CORBA.portable.ValueInputStream defines methods that allow deserialization of custom-marshaled RMI/IDL objects from GIOP streams. // Java package org.omg.CORBA.portable; public interface ValueInputStream { void start_value(); void end_value(); } 10. In existing section 1.5.1.3, add a new paragraph at the end of the "Execution Model for Serialization" subsection as follows: The ORB output stream passed to the ValueHandlerMultiFormat.writeValue method must implement the ValueOutputStream interface, and the ORB input stream passed to the ValueHandler.readValue method must implement the ValueInputStream interface. For output streams, the start_value() method ends any currently open chunk, writes a valuetype header for a nested custom valuetype (with a null codebase and the specified repository ID), and increments the valuetype nesting depth. The end_value() method ends any currently open chunk, writes the end tag for the nested custom valuetype, and decrements the valuetype nesting depth. For input streams, the start_value() method reads a valuetype header for a nested custom valuetype and increments the valuetype nesting depth. The end_value() method reads the end tag for the nested custom valuetype (after skipping any data that precedes the end tag) and decrements the valuetype nesting depth. 11. In existing section 1.5.1.3, add new paragraphs at the end of the "Value Marshaling" subsection as follows: Before calling the writeValue method of the ValueHandler object, the ORB must determine the stream format version to be used. This is the maximum format version that is supported by both the local ValueHandler and the remote connection endpoint. The maximum local format version is the value returned by the getMaximumStreamFormatVersion method of the ValueHandler object, or 1 if the ValueHandler object doesn't support the ValueHandlerMultiFormat interface. The maximum remote format version is 1 for GIOP 1.2 messages and 2 for GIOP 1.3 messages, except where these default values are overridden by either the TAG_RMI_CUSTOM_MAX_STREAM_FORMAT tagged component (see section 1.4.11) or the RMICustomMaxStreamFormat service context (see section 1.4.12). For GIOP 1.2 messages, recognition of these overrides is optional. If the stream format version computed in this way is 2 or greater, the ORB must call the ValueHandlerMultiFormat.writeValue method, passing this value. If the stream format version computed in this way is 1, the ORB may call either the ValueHandlerMultiFormat.writeValue method (with stream format 1) or the ValueHandler.writeValue method. If the ORB's call to the ValueHandler object's writeValue method specified RMI/IDL custom value type stream format version 2, then the ValueHandler object must call the ValueOutputStream.start_value() and ValueOutputStream.end_value() methods of the ORB stream before and after writing the data specified by item 1d of section 1.4.10. The rep_id string passed to the start_value() method must be "RMI:org.omg.custom.::" where is the fully-qualified name of the class whose writeObject method is being invoked and and are the class's hashcode and SUID. For format version 2, if the ORB stream passed to the ValueHandler doesn't support the ValueOutputStream interface, then a BAD_PARAM exception with standard minor code (note to editor: number to be assigned by OMG) must be thrown. 12. In existing section 1.5.1.3, add a new paragraph at the end of the "Value Unmarshaling" section as follows: If the RMI/IDL custom data unmarshaled from the input stream was encoded using stream format 2, then the ValueHandler object must call the ValueInputStream.start_value() and ValueInputStream.end_value() methods of the ORB stream before and after reading the data specified by item 1d of section 1.4.10. For format version 2, if the ORB stream passed to the ValueHandler doesn't support the ValueInputStream interface, then a BAD_PARAM exception with standard minor code (note to editor: number to be assigned by OMG) must be thrown. If the format version unmarshaled by the ValueHandler is greater than the maximum version that it supports, then a MARSHAL exception with standard minor code
(note to editor: number to be assigned by OMG) must be thrown. and the changes needed to the CORBA specification, document number formal/2001-09-34: 13. In section 4.11.4, add the following exception standard minor codes: BAD_PARAM (note to editor: number to be assigned by OMG) Unsupported RMI/IDL custom value type stream format. BAD_PARAM (note to editor: number to be assigned by OMG) ORB output stream does not support ValueOutputStream interface. BAD_PARAM (note to editor: number to be assigned by OMG) ORB input stream does not support ValueInputStream interface. MARSHAL
(note to editor: number to be assigned by OMG) Unsupported RMI/IDL custom value type stream format. 14. In section 13.6.6, add the following ComponentId constant definition to the first list: const ComponentId TAG_RMI_CUSTOM_MAX_STREAM_FORMAT = xx; // number to be assigned by OMG const ServiceId RMICustomMaxStreamFormat = xx; // number to be assigned by OMG 15. In section 13.6.6.3, add the following bullet: 16. In section 13.7.1, add the following ServiceId constant definition: const ServiceId RMICustomMaxStreamFormat = xx; // number to be assigned by OMG and the following bullet: described in the Java to IDL mapping specification (formal/xx-xx-xx), section 1.4.12 "RMICustomMaxStreamFormat Service Context". 17. In section 13.9, add a row to the table for feature "RMICustomMaxStreamFormat service context" showing that it is optional for version 1.2 and mandatory for GIOP 1.3 (when the 1.3 column is added).. 18. In section A.2, add a row to the table for ServiceId tag value RMICustomMaxStreamFormat indicating that it is described in the Java to IDL mapping specification (formal/xx-xx-xx), section 1.4.12. 19. In section A.3, add a row to the table for ComponentId tag value TAG_RMI_CUSTOM_MAX_STREAM_FORMAT indicating that it is described in the Java to IDL mapping specification (formal/xx-xx-xx), section 1.4.11. Simon -- Simon C Nash, Chief Technical Officer, IBM Java Technology Tel. +44-1962-815156 Fax +44-1962-818999 Hursley, England Internet: nash@hursley.ibm.com Lotus Notes: Simon Nash@ibmgb s more B-data to be written, while everything else stays the same. When the old B code unmarshals a new B instance, it will not fully consume all the B-data and this will cause the unmarshalling of the C-data to be incorrect since the stream will be out of sync. The problem is caused by a deficiency in the stream encoding for custom serialized data. In the example above, the existing format does not identify the boundary between the B-data and the C-data, so it is impossible for the unmarshalling code in the ValueHandler to skip over any extraneous B-data before it starts to read the C-data. Having carefully considered a number of possible solutions to this problem, I think Bob Scheifler's proposal of enclosing the B-data in a nested valuetype is the best approach. The repid for the nested valuetype must be specified by the Java to IDL mapping so that it can be read by IDL custom unmarshalling code in other languages. I propose that this repid be "RMI:org.omg.custom.::" where is the fully-qualified name of the class whose writeObject method is being invoked and and are the class's hashcode and SUID. The other language will have to provide a suitable custom valuetype implementation for the nested valuetype and may have to provide a value factory that maps the standard repid to this implementation. For Java unmarshalling, the ValueHandler will understand the special repid and process its contents as RMI-IIOP custom serialized data. Since valuetype headers and end tags are written by the ORB's OutputStream and not by the ValueHandler, it will be necessary to add new APIs to the OutputStream for the ValueHandler to call when marshalling custom valuetype data. These new APIs will write the valuetype header and end tag for the nested valuetype, keep track of the valuetype nesting level, and take care of chunking for the nested valuetype. Rather than adding yet another level of OutputStream subclassing, I propose adding the new APIs to an interface called org.omg.CORBA.portable.ValueOutputStream which will be implemented by the output stream passed to the ValueHandler. The new interface is defined as follows: public interface ValueOutputStream { void start_value(java.lang.String rep_id); void end_value(); } When the ValueHandler encounters a class with custom serialized data, it calls the start_value() method on the output stream that it was given, passing the special custom repid for the custom-serialized class. The output stream ends the currently open chunk, writes a valuetype header for the nested custom valuetype, and increments the valuetype nesting depth. It then returns to the ValueHandler, which calls writeObject() to write the custom serialized data. When writeObject() returns, the ValueHandler calls end_value() on the stream to end the current chunk, write the end tag, and decrement the valuetype nesting depth. When unmarshalling a class with custom serialized data, the ValueHandler needs to call the ORB's input stream to consume the valuetype header and end tag. A new interface org.omg.CORBA.portable.ValueInputStream that defines the necessary APIs will be implemented by the input stream. This interface is defined as follows: public interface ValueInputStream { void start_value(); void end_value(); } The ValueHandler calls the start_value() method on the input stream. The input stream reads the nested valuetype header and increments the valuetype nesting depth, then returns to the ValueHandler. The ValueHandler reads the custom serialized data, then calls end_value() which consumes the end tag (after skipping any data that precedes the end tag) and decrements the valuetype nesting depth. For languages other than Java, the nested valuetype must be handled explicitly by calling write_Value() and read_Value() within the implementations of the marshal() and unmarshal() methods. To assist in the implementation of these methods, a new IDL valuetype is generated by the Java to IDL mapping. In the example above, this would be: module org { module omg { module customRMI { custom valuetype B {}; #pragma ID B "RMI:org.omg.customRMI.B:1234567890ABCDEF:FEDCBA0987654321" }; }; }; This would be implemented in C++ (for example) with marshal() and unmarshal() methods that read and write the appropriate data. The new stream format would be identified by a format version of 2 (currently 1). By default, ORBs that support GIOP version 1.2 are assumed to accept format version 1 only, and ORBs that support GIOP 1.3 are assumed to accept format versions 1 and 2. These defaults can be overridden by a new tagged component for servers, and by a new service context for clients. The new tagged component is identified by the following component id: const ComponentId TAG_RMI_CUSTOM_MAX_STREAM_FORMAT = xx; // number to be assigned by OMG The TAG_RMI_CUSTOM_MAX_STREAM_FORMAT component has an associated value of type octet, encoded as a CDR encapsulation, designating the maximum stream format version for RMI/IDL custom value types that can be used in GIOP messages sent to this IOR. The TAG_RMI_CUSTOM_MAX_STREAM_FORMAT component can appear at most once in any IOR profile. For profiles supporting IIOP 1.2 or greater, it is optionally present. If this component is omitted, then the default maximum stream format version for RMI/IDL custom value types sent to this IOR is 1 for IIOP 1.2 and 2 for IIOP 1.3. The new service context is identified by the following service id: const ServiceId RMICustomMaxStreamFormat = xx; // number to be assigned by OMG RMICustomMaxStreamFormat identifies a CDR encapsulation of a single octet that specifies the highest RMI/IDL custom stream format version that can be used for RMI/IDL custom valuetypes marshaled within a GIOP reply associated with the GIOP request that carries this service context. Here's a formal proposal for the changes needed to the Java to IDL mapping spec, document number ptc/2001-06-15: 1. In section 1.3.5.6, add the following at the end of the second paragraph: An additional IDL custom valuetype in the module ::org::omg::customRMI is also generated to assist with marshaling and unmarshaling instances of the class. See section 1.3.5.8 for details. 2. Add a new section 1.3.5.8 "Secondary custom valuetype", renumbering the existing section 1.3.5.8 and subsequent sections: In addition to the primary mapping described above, an RMI/IDL value type containing a writeObject method is mapped to a secondary IDL custom valuetype. The module name for this valuetype is formed by taking the ::org::omg::customRMI prefix and then adding the primary mapped type