Issue 1407: Interaction of find_POA() and AdapterActivatorts and create_POA() (port-rtf) Source: (, ) Nature: Uncategorized Issue Severity: Summary: Summary: 1. Interaction of find_POA() and AdapterActivators and create_POA(). In a single-threaded environment, one can call find_POA() and cause an AdapterActivator to be invoked, which will ultimately call create_POA() and return control back to the user after completeing the initialization of that POA. In a multi-threaded environment, things are not that simple. One can call find_POA(), which will invoke an AdapterActivator. Now from here many things can happen. For instance, another thread can come in and call create_POA() and create the POA before the AdapterActivator gets a chance to. The AdapterActivator will then need to handle this unexpected condition. Granted, this was probably a poorly written program, but properly designed APIs should still have defined behavior for even incorrectly written programs. Resolution: Revised Text: Actions taken: June 1, 1998: received issue July 30, 1998: closed issue Discussion: End of Annotations:===== Return-Path: Sender: gscott@inprise.com Date: Fri, 29 May 1998 11:45:09 -0700 From: "George M. Scott" Organization: Borland International, Inc. To: issues@omg.org CC: port-rtf@omg.org Subject: POA lifecycle issues in MT environments POA lifecycle (creation and destruction) as defined currently has lots of potential problem in multi-threaded environments. I have outlined a couple of the problems below. We believe the semantics should be clarified so POA-based servers will be portable in MT environments. 1. Interaction of find_POA() and AdapterActivators and create_POA(). In a single-threaded environment, one can call find_POA() and cause an AdapterActivator to be invoked, which will ultimately call create_POA() and return control back to the user after completeing the initialization of that POA. In a multi-threaded environment, things are not that simple. One can call find_POA(), which will invoke an AdapterActivator. Now from here many things can happen. For instance, another thread can come in and call create_POA() and create the POA before the AdapterActivator gets a chance to. The AdapterActivator will then need to handle this unexpected condition. Granted, this was probably a poorly written program, but properly designed APIs should still have defined behavior for even incorrectly written programs. Proposed solution: In multi-threaded environments a call to find_POA() which requires an AdapterActivator to be invoked, must prevent any other thread from calling create_POA() until the AdapterActivator returns. This also applies to requests received over-the-wire which cause the invocation of AdapterActivators. Threads which do attempt to call create_POA() will block until the thread invoking the AdpaterActivator returns. Note, this also implies that all calls to AdapterActivators are now serialized, which is a change in behavior from the current spec. With this solution, and the above MT scenario, the AdapterActivator would complete normally as it would have in a single-threaded environment, and the thread att Return-Path: Date: Thu, 04 Jun 1998 19:56:29 -0400 From: Paul H Kyzivat Organization: NobleNet To: "George M. Scott" CC: issues@omg.org, port-rtf@omg.org Subject: Re: POA lifecycle issues in MT environments References: <356F0235.F57BE98F@inprise.com> George M. Scott wrote: > > POA lifecycle (creation and destruction) as defined currently has > lots > of potential problem in multi-threaded environments. I have > outlined > a couple of the problems below. We believe the semantics should be > clarified so POA-based servers will be portable in MT environments. Sorry to take so long before replying, but I was too busy to think through a sufficient reply. > > 1. Interaction of find_POA() and AdapterActivators and create_POA(). > > In a single-threaded environment, one can call find_POA() and cause > an AdapterActivator to be invoked, which will ultimately call > create_POA() > and return control back to the user after completeing the > initialization of > that POA. > > In a multi-threaded environment, things are not that simple. One > can > call > find_POA(), which will invoke an AdapterActivator. Now from here > many > things can happen. For instance, another thread can come in and > call > create_POA() > and create the POA before the AdapterActivator gets a chance to. > The > AdapterActivator will then need to handle this unexpected condition. > Granted, this was probably a poorly written program, but properly > designed > APIs should still have defined behavior for even incorrectly written > programs. > > Proposed solution: In multi-threaded environments a call to > find_POA() which > requires an AdapterActivator to be invoked, must prevent any other > thread > from calling create_POA() until the AdapterActivator returns. This > also applies > to requests received over-the-wire which cause the invocation of > AdapterActivators. > Threads which do attempt to call create_POA() will block until the > thread invoking > the AdpaterActivator returns. Note, this also implies that all > calls > to AdapterActivators > are now serialized, which is a change in behavior from the current > spec. I think you have too much concern for the inept developer. The kind of locking that your proposal implies is a lot of trouble for no particularly good reason. It implies a whole new set of machinery to block requests ON POAs, distinct from the that needed to block requests on objects managed by POAs A developer writing a threaded program should be prepared for the case you describe. There is however a problem here because of the odd design of the unknown_adapter operation. Suppose that the AdapterActivator decides not to create the POA and returns FALSE. Now suppose that at the same time some other thread creates the POA. When unknown_adapter returns, is the now-present POA used or not? The problem is that the boolean return value from unknown_adapter is useless. If the return is TRUE, the caller still must check for the existence of the POA, because the TRUE return may have been a lie. At best a FALSE return could be treated as a veto over any POA that might be present then, but this example illustrates that doing so may lead to race conditions. I believe the boolean return value from this operation should be eliminated; and until it can be, it should be documented as being irrelevant. Otherwise I don't think the problem George presents requires a solution. > 2. POA destroy() operation ill-defined. > > The POA destroy() operation is ill-defined with respect to what > happens > to requests which are currently executing when destroy() is called. > > Currently, the spec states: > > "When a POA is destroyed, any requests that have started execution > continue > to completion. Any requests that have not started execution are > processed as if > they were newly arrived, that is, the POA will attempt to cause > recreation of the > POA by invoking one or more adapter activators." > > If there is a current thread executing in some Servant in some POA > and > another > thread invokes destroy(), and immediately thereafter a new request > arrives for the > same POA there are many possibilities I can envision from reading > the > spec. > There should be only one possible behavior in this situation. > > Let's look at this in more detail: > > If the thread invoked destroy(false, false), meaning do not perform > etherialization and do > not wait for completion, the spec states destroy() should just > "destroy" the POA and > return. In this case the newly arrived request will invoke an > AdapterActivator and > create a new POA with the same name. But, now there are two POAs > with > the same > name running in the same process! What should happen if the thread > invoking the > servant in the original POA, executes the following code: > > // Java > // Note in the Java mapping _poa() is an operation on Servant which > returns > // the same as POACurrent::get_poa(). > > _poa().the_parent().find_POA(_poa().the_name(), > false).servant_to_reference(this); > > This code goes to the parent POA, looksup the current POA's poa name > and then > performs a servant_to_reference() on itself. The trouble is that > the > parent has removed > the current POA from its internal tables and replaced it with the > new > POA which was > created due to the recently received request. Since this servant > was > not activated in the > new POA, this operation will actually fail! > > Note, the scenario I describe is not that far-fetched, and in an > exteme case could result > in many many copies of the same POA existing in the same process. > > Discussion: It should be impossible to have two POAs running with > the > same name at the > same time even if all but one are in a semi-destoryed state. POAs > should not exist in some > semi-destroyed state, while activations continue to run. Agreed. > > Solution: Change behavior of destroy() as follows: > > Change second paragraph to the following: > > Before a POA is destroyed, any requests that have started execution > continue to completion. > Any requests that have not started execution are queued until POA > destruction is complete > and then behave as if they were newly arrived, that is, the POA will > attempt to cause > recreation of the POA by invoking one or more adapter activators. I think this is reasonable. > > Change fourth paragraph to the following: > > If the wait_for_completion paramter is TRUE, the destroy operation > will return only after > all requests in process have completed and if all inovcations of > etherialize have completed > (if etheralize_objects is also TRUE). If destroy is invoked on > some > POA A from a thread > which is currently executing a request in some POA B, where B is > either A or some descendant > of A, then destroy will return after all of B's descandants have > been > destroyed and all other > requests in A have completed, remaining descandants of A and A > itself > will be queued for > destruction. If wait_for_completion is FALSE, destroy operations > queues the POA for > destruction when all requests have completed. > > [I'll admit the above paragraph is somewhat confusing and needs some > work]. > > What I'm trying to say is that if there is some POA hierarchy > > root->A->B->C > > And a thread is invoking some servant in B which calls > A.destroy(false, true). That the > call will block until C is destroyed and that all other requests > (not > including the thread > invoking destroy()) have completed in poa B, at which point the call > will return and queue > A and B for destruction by another thread. > > Other issues: There is a similar issue for etherealization when > destroy is invoked by > a thread executing in a Servant which needs to be etherialzed. I > believe there has been > some discussion on this, so I wanted to go back and check the > archvies > to see what thoughts > there were on this. > Some of these issues were subsequently reposted in different form around how to handle WAIT_FOR_COMPLETION. That was a better context to discuss them. You perhaps hint at another problem. If WAIT_FOR_COMPLETION is FALSE, there are significant problems in guaranteeing that the POA has been destroyed when the call returns without forcing an implicit wait. The problem is how to deal with subordinate POAs. They must also be destroyed, and if they have ServantActivators and etherealization is specified, then those activators will need to be invoked. They may well be activated in one of their ancestor POAs, so the ancestors had better not be destroyed before their etherializations are complete. The most significant example of this comes in the case of destroying the root POA, either explicitly, or implicitly through an ORB shutdown. I believe the best that can be done is to say that, when destroy is called on a POA with wait_for_completion false and etherealize true, on return from the call a series of events will have been initiated that will eventually lead to the POA being destroyed, but that it may remain active for some period of time. In some cases, especially with single threaded ORBs, it may also require calls to ORB::perform_work before the POA will be completely destroyed. > There is also a problem with multiple threads calling destroy() once a > destroy() operation has > begun. This is particularly important when they set > wait_for_completion to true, or if they > call destroy with a different etherealize_objects value. My current > thought on this is that > the first thread which calls destroy determines the etherealization > policy for the destroy. > If subsequent threads call destroy() while destruction is in progress > they will wait for completion > if wait_for_completion is true, otherwise they return immediately, and > let the intial thread which > called destroy() complete the destruction. This is also reasonable. Return-Path: Sender: gscott@inprise.com Date: Fri, 05 Jun 1998 14:29:08 -0700 From: "George M. Scott" Organization: Borland International, Inc. To: Paul H Kyzivat CC: issues@omg.org, port-rtf@omg.org Subject: Re: POA lifecycle issues in MT environments References: <356F0235.F57BE98F@inprise.com> Paul H Kyzivat wrote: > George M. Scott wrote: > > > > > 1. Interaction of find_POA() and AdapterActivators and create_POA(). > > > > In a single-threaded environment, one can call find_POA() and cause > > an AdapterActivator to be invoked, which will ultimately call > > create_POA() > > and return control back to the user after completeing the > > initialization of > > that POA. > > > > In a multi-threaded environment, things are not that simple. One can > > call > > find_POA(), which will invoke an AdapterActivator. Now from here many > > things can happen. For instance, another thread can come in and call > > create_POA() > > and create the POA before the AdapterActivator gets a chance to. The > > AdapterActivator will then need to handle this unexpected condition. > > Granted, this was probably a poorly written program, but properly > > designed > > APIs should still have defined behavior for even incorrectly written > > programs. > > > > Proposed solution: In multi-threaded environments a call to > > find_POA() which > > requires an AdapterActivator to be invoked, must prevent any other > > thread > > from calling create_POA() until the AdapterActivator returns. This > > also applies > > to requests received over-the-wire which cause the invocation of > > AdapterActivators. > > Threads which do attempt to call create_POA() will block until the > > thread invoking > > the AdpaterActivator returns. Note, this also implies that all calls > > to AdapterActivators > > are now serialized, which is a change in behavior from the current > > spec. > > I think you have too much concern for the inept developer. The kind of > locking that your proposal implies is a lot of trouble for no > particularly good reason. It implies a whole new set of machinery to > block requests ON POAs, distinct from the that needed to block requests > on objects managed by POAs Having written the code to handle the spec as it currently stands and the specwith our propsed changes I can tell you that the code with our proposed changes in considerably simpler than the code to implement the current spec. The current spec requires complex queueing on a per-POA basis until the AdapterActivator returns. In the proposed model, there can only be one activation outstanding at a time, so a simple recursive mutex can be used. > There is however a problem here because of the odd design of the > unknown_adapter operation. Suppose that the AdapterActivator decides > not > to create the POA and returns FALSE. Now suppose that at the same > time > some other thread creates the POA. When unknown_adapter returns, is > the > now-present POA used or not? > > The problem is that the boolean return value from unknown_adapter is > useless. If the return is TRUE, the caller still must check for the > existence of the POA, because the TRUE return may have been a > lie. At > best a FALSE return could be treated as a veto over any POA that > might > be present then, but this example illustrates that doing so may lead > to > race conditions. > > I believe the boolean return value from this operation should be > eliminated; and until it can be, it should be documented as being > irrelevant. I agree with this pretty much. The boolean return value is pretty meaningless.It would have been preferable to at least return a POA reference or null if the Activator failed. Even this has some of the problems you mentioned, however. It is interesting to note that one of the primary reasons the AdapterActivators cause requests for the new POA to block while it completes is so that it can properly initialize the POA with servant managers, activators, default servants, etc. and avoid race conditions I mentioned in issue 1410. However, if the change to issue 1410 is accepted, then this should not be a problem since a properly written AdapterActivator would have to create a new POA with a manager in non-active state in order to set the servant managers, etc. so as soon as create_POA() is called that POA can be made immediately visible, though it may be in a non-active state if it is currently being initialized. George Return-Path: Date: Fri, 05 Jun 1998 20:05:14 -0400 From: Paul H Kyzivat Organization: NobleNet To: "George M. Scott" CC: issues@omg.org, port-rtf@omg.org Subject: Re: POA lifecycle issues in MT environments References: <356F0235.F57BE98F@inprise.com> <35786324.68B5F3EB@inprise.com> George M. Scott wrote: > > Paul H Kyzivat wrote: > > > George M. Scott wrote: > > > > > > > > 1. Interaction of find_POA() and AdapterActivators and > create_POA(). > > > > > > In a single-threaded environment, one can call find_POA() and > cause > > > an AdapterActivator to be invoked, which will ultimately call > > > create_POA() > > > and return control back to the user after completeing the > > > initialization of > > > that POA. > > > > > > In a multi-threaded environment, things are not that simple. > One > can > > > call > > > find_POA(), which will invoke an AdapterActivator. Now from > here > many > > > things can happen. For instance, another thread can come in and > call > > > create_POA() > > > and create the POA before the AdapterActivator gets a chance to. > The > > > AdapterActivator will then need to handle this unexpected > condition. > > > Granted, this was probably a poorly written program, but > properly > > > designed > > > APIs should still have defined behavior for even incorrectly > written > > > programs. > > > > > > Proposed solution: In multi-threaded environments a call to > > > find_POA() which > > > requires an AdapterActivator to be invoked, must prevent any > other > > > thread > > > from calling create_POA() until the AdapterActivator returns. > This > > > also applies > > > to requests received over-the-wire which cause the invocation of > > > AdapterActivators. > > > Threads which do attempt to call create_POA() will block until > the > > > thread invoking > > > the AdpaterActivator returns. Note, this also implies that all > calls > > > to AdapterActivators > > > are now serialized, which is a change in behavior from the > current > > > spec. > > > > I think you have too much concern for the inept developer. The > kind > of > > locking that your proposal implies is a lot of trouble for no > > particularly good reason. It implies a whole new set of machinery > to > > block requests ON POAs, distinct from the that needed to block > requests > > on objects managed by POAs > > Having written the code to handle the spec as it currently stands > and > the specwith our propsed changes I can tell you that > the code with our proposed changes > in considerably simpler than the code to implement the current spec. > The current > spec requires complex queueing on a per-POA basis until the > AdapterActivator > returns. In the proposed model, there can only be one activation > outstanding at > a time, so a simple recursive mutex can be used. Having also written such code, I don't see it that way. As it stands, it is only necessary to serialize calls by a POA to its AdapterActivator - requiring a simple mutex. With the proposed change, some subset of the explicitly callable operations on the POA must be blocked under this particular circumstance. This feels like it might be the first step down a slippery slope, leading to a bunch of increasingly complex and somewhat arbitrary rules. If I fully understand your proposal (probably not) you seem to suggest a global block on the use of create_POA, for any POA for any purpose, not just of the particular POA being activated. This is potentially a bottleneck on unrelated activities in the server. It would also prevent the possibility that the thread that receives the unknown_adapter call might hand the actual work of creating the POA off to another thread. If you want to implement it this way, perhaps you could propose a change to the spec that would permit such an implementation without requiring it. > > > There is however a problem here because of the odd design of the > > unknown_adapter operation. Suppose that the AdapterActivator > decides > not > > to create the POA and returns FALSE. Now suppose that at the same > time > > some other thread creates the POA. When unknown_adapter returns, > is > the > > now-present POA used or not? > > > > The problem is that the boolean return value from unknown_adapter > is > > useless. If the return is TRUE, the caller still must check for > the > > existence of the POA, because the TRUE return may have been a lie. > At > > best a FALSE return could be treated as a veto over any POA that > might > > be present then, but this example illustrates that doing so may > lead > to > > race conditions. > > > > I believe the boolean return value from this operation should be > > eliminated; and until it can be, it should be documented as being > > irrelevant. > > I agree with this pretty much. The boolean return value is pretty > meaningless.It would have been preferable to at least > return a POA reference or null if > the Activator failed. Even this has some of the problems you > mentioned, however. Do you think it is worth trying to change the spec about this, or shall we just leave well enough alone and have each ORB implementor figure out that this should just be ignored? > > It is interesting to note that one of the primary reasons the > AdapterActivators > cause requests for the new POA to block while it completes is so > that > it can > properly initialize the POA with servant managers, activators, > default > servants, > etc. and avoid race conditions I mentioned in issue 1410. > > However, if the change to issue 1410 is accepted, then this should > not > be a > problem since a properly written AdapterActivator would have to > create > a new POA with a manager in non-active state in order to set the > servant > managers, etc. so as soon as create_POA() is called that POA can be > made > immediately visible, though it may be in a non-active state if it is > currently > being initialized. The developer already has the option of creating the new POA with a distinct POAManager and then activating it only when ready. But the change would in effect mandate that every POA must have its own manager. If a server wants to share a POAManager among a number of POAs, and also wants to dynamically activate those POAs, then the proposal would mean that all the POAs already in existence would have to be blocked for the duration while another member of the family was activated. That could be a pretty serious impact on performance. Return-Path: Sender: "George Scott" Date: Wed, 01 Jul 1998 19:01:35 -0700 From: "George M. Scott" Organization: Inprise Corporation To: port-rtf@omg.org Subject: Proposed resolutions for Issues 1407-1410, 1428 The following are the proposals to address issues 1407-1410, and 1428. The proposals are based on our discussions from the previous POA conference call, and should hopefully be acceptable. I'm attempting to follow Dan's format for consistency. George ----------------------------------------------------------- Issue 1407: Interaction of find_POA(), AdapterActivators and create_POA() Nature: Clarification Summary: There is a race condition in the POA when an AdapterActivator is invoked as a result of a call to find_POA() or an over-the-wire request, and simultaneously another thread calls create_POA() attempting to create the same POA manually. Resolution: Accepted for Corba RTF 2.3 Revision: Section 9.3.3, Page 9-20. After first paragraph in page beginning with "if unknown_adapter raises..." add the following Note: Note - It is possible for another thread to create the same POA the AdapterActivator is being asked to create if AdapterActivators are used in conjunction with other threads calling create_POA with the same POA name. To avoid potential race conditions, it is recommended that AdapterActivators and manual creation of POAs (via the create_POA call) not be used in conjunction to create a particular POA. Return-Path: Date: Mon, 06 Jul 1998 13:55:46 -0400 From: Paul H Kyzivat Organization: NobleNet To: "George M. Scott" CC: port-rtf@omg.org Subject: Re: Proposed resolutions for Issues 1407-1410, 1428 References: <359AE9FF.441D15F7@inprise.com> George M. Scott wrote: > > The following are the proposals to address issues 1407-1410, > and 1428. The proposals are based on our discussions from > the previous POA conference call, and should hopefully be > acceptable. > > I'm attempting to follow Dan's format for consistency. > > George > > ----------------------------------------------------------- > Issue 1407: Interaction of find_POA(), AdapterActivators and > create_POA() > Nature: Clarification > > Summary: There is a race condition in the POA when an > AdapterActivator is invoked as a result of a > call to find_POA() or an over-the-wire request, and > simultaneously another thread calls create_POA() > attempting to create the same POA manually. > > Resolution: Accepted for Corba RTF 2.3 > > Revision: Section 9.3.3, Page 9-20. After first paragraph > in page beginning with "if unknown_adapter raises..." > add the following Note: > > Note - It is possible for another thread to create the same > POA the AdapterActivator is being asked to create if > AdapterActivators are used in conjunction with other threads > calling create_POA with the same POA name. To avoid potential > race conditions, it is recommended that AdapterActivators and > manual creation of POAs (via the create_POA call) not be used > in conjunction to create a particular POA. > > ------------------------------------------------------------- > Issue 1408: POA destory() is ill-defined > Issue 1409: Multiple threads calling destroy() once destroy() > has begun > Nature: Revision > > Summary: POA destroy is not defined sufficiently enough to > prevent multiple activations of the same POA name in > the same process. POA::destory semantics are not defined > for multiple threads calling destroy. > > Resolution: Accepted for Corba RTF 2.3 > > Revision: The following text will replace the current describing > the destroy operation in Section 9.3.8, Page 9-31. > > This operation destroys the POA and all descendant POAs. > Descendant POAs are destroyed (recursively) before the destruction > of the containing POA. The POA so destroyed (that is, the POA > with its name) may be re-created later in the same process. > (This differs from the POAManager::deactivate operation that does > not allow a re-creation of its associated POA in the same process. > > When a POA is destroyed, any requests that have started execution > continue to completion. Any requests that have not started > execution are processed as if the POA were in the holding state, > until execution of all active requests has completed. Once > all active requests have completed, all queued requests (if > any) will behave as if they were newly arrived, that is, the POA > will attempt to cause recreation of the POA by invoking one or > more adapter activators. > > If the etherealize_objects parameter is TRUE, the POA has the > RETAIN policy, and a servant manager is registered with the POA, > the etherealize operation on the servant manager will be called > for each active object in the Active Object Map. Etherealization > will not occur until all active requests have completed execution. > The apparent destruction of the POA occurs before any calls > to etherealize are made. Thus, for example, an etherealize > method that attempts to invoke operations on the POA will > receive the OBJECT_NOT_EXIST exception. > > {editorial note - below I have merged text from these two issues > and issue 1428 because they change the same text. If > one of these resolutions fails, we will need to edit the following > text) > > The wait_for_completion parameter is handled as follows: > > - If wait_for_completion is TRUE and the current > thread is not in a POA-dispatched invocation context, the > destroy operation will return only after all active requests > have completed and all invocations of etherealize have > completed. > > - If wait_for_completion is TRUE and the current thread is in a > POA-dispatched invocation context, then the BAD_INV_ORDER > exception is thrown and POA destruction does not occur. > > - If wait_for_completion is FALSE, the destroy operation > destroys all POAs but does not wait for active requests to > complete nor for etherealization to occur. > > If destroy is called multiple times before destruction is > complete (because there are active requests), the > etherealize_objects parameter will only apply to the first > call of destroy, subsequent calls with conflicting > etherealize_objects settings will use the value of the > etherealize_objects from the first call. The wait_for_completion > parameter will be handled as defined above for each individual > call (some callers may choose to block, while others may not). > This seems reasonable. > -------------------------------------------------------------- > Issue 1410: Changing default servant, etc. after POA activation > Nature: Revision > > Summary: Changing default servants, servant managers, etc. > after POA activation can be problematic. After some > discussion it was agreed that this is only truly > problematic for ServantLocators which need to have > matching preinvoke/postinvoke operations. Failing > to do so could result in memory leaks (due to the > Cookie parameter) or other strange side effects, > including inproper transaction management. > > Resolution: Accepted for Corba 2.3 RTF > > Revision: There are two current proposed revisions, which are > outlined below. Inprise will vote for proposal (a) > over proposal (b). > > (a) > Change the second paragraph describing the set_servant_manager > operation on page 9-33, Section 9.3.8 to the following: > > "This operation sets the default servant manager associated > with the POA. This operation may only be invoked once after > a POA has been created. Attempting to set the servant manager > after one has already been set will result in the BAD_INV_ORDER > exception being thrown." I find this overly restrictive. > > (b) > Add a third paragraph to the description of set_servant_manager > on page 9-33, Section 9.3.8: > > "Changing a ServantLocator will result in newly received requests > calling preinvoke and postinvoke on the new ServantLocator. > However, currently active requests will call postinvoke on the > ServantLocator on which they called preinvoke, which may not be > the same as the current ServantLocator." I also find this overly restrictive. It imposes a particular implementation discipline to cover an obscure case that will rarely arise in practice and which a developer can work around if it does arise. I would assert: - in most cases there will be no need to change the locator, - when it is necessary, the developer will probably know that the change is safe, - when the developer doesn't know it is safe, the POAManager state can be set to holding before making the change to guarantee safety. So, I don't think any change is needed, except for some words of warning, replacing the ones above: "If a ServantLocator is replaced while operations are outstanding, then the old ServantLocator may be called for preinvoke and the replacement for postinvoke of the same request. If this is unacceptable, then the POA's POAManager may be placed in the HOLDING state prior to making the change." > > ---------------------------------------------------------------- > Issue 1428: Blocking POA Operations > Nature: Revision > > Summary: Several operations added to CORBA as part of the > Portability submission provide blocking behavior which > can result in deadlock in a large number of cases. > These calls include POA::destroy, ORB::shutdown, > POAManager::deactivate, POAManager::hold_requests, > POAManager::discard_requests. > > Resolution: Accepted for Corba 2.3 RTF > > Revision: The following changes are proposed: > > Replace the second sentence in the paragraph of section 4.9.4, > page 4-20, which begins with "If the wait_for_completion ..." > with the following: > > "If the wait_for_completion parameter is TRUE and the current > thread is not in an ORB-dispatched invocation context, this > operation blocks until all ORB processing (including request > processing and object deactivation or other operations associated > with object adapters) has completed. If the wait_for_completion > parameter is TRUE and the current thread is in an ORB-dispatched > invocation context, then the BAD_INV_ORDER exception is thrown." > > Replace the phrase "If the parameter is TRUE" in the 2nd > paragraph, 2nd sentence of the hold_requests description in > Section 9.3.2, page 9-18, with the phrase "If the parameter > is TRUE and the current thread is not in a POA-dispatched > invocation context". Add the sentenence "If the parameter > is TRUE and the current thread is in a POA-dispatched invocation > context then the BAD_INV_ORDER exception is raised and the > state is not changed." > > Replace the phrase "If the parameter is TRUE" in the 2nd > paragraph, 2nd sentence of the discard_requests description in > Section 9.3.2, page 9-18, with the phrase "If the parameter > is TRUE and the current thread is not in a POA-dispatched > invocation context". Add the sentenence "If the parameter > is TRUE and the current thread is in a POA-dispatched invocation > context then the BAD_INV_ORDER exception is raised and the > state is not changed." to the end of the paragraph. > > Replace the phrase "If the parameter is TRUE" in the 3rd > paragraph, 2nd sentence of the deactivate description in > Section 9.3.2, page 9-18, with the phrase "If the parameter > is TRUE and the current thread is not in a POA-dispatched > invocation context". Add the sentenence "If the parameter > is TRUE and the current thread is in a POA-dispatched invocation > context then the BAD_INV_ORDER exception is raised and the > state is not changed." to the end of the paragraph. > > (editorial note- I changed the text for POA::destory in the > resolution of issues 1408 and 1409 above, so it is not shown > here but a similar change to the above is necessary. > I also felt that 1409 raised an issue which also affects > POAManager::deactivate, so the following change is also > proposed) > > Add the following paragraph to the end of the description of > deactivate in section 9.3.2, page 9-19. > > "If deactivate is called multiple times before destruction is > complete (because there are active requests), the > etherealize_objects parameter will only apply to the first > call of deactivate, subsequent calls with conflicting > etherealize_objects settings will use the value of the > etherealize_objects from the first call. The wait_for_completion > parameter will be handled as defined above for each individual > call (some callers may choose to block, while others may not)." This seems reasonable Return-Path: From: "Daniel R. Frantz" To: Subject: Issue 1407 (port-rtf) Slightly amended proposal. Please vote. Date: Wed, 8 Jul 1998 09:48:53 -0400 X-MSMail-Priority: Normal Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V4.72.2106.4 At the conference call yesterday, we decided that the words of 1407 should be slightly amended. The words "race condition" didn't seem quite right to some people so we've found a new way to express the resolution. Voters, even if you voted on the previous version, please re-vote on this version. Dan ==================================================================== Issue 1407: Interaction of find_POA(), AdapterActivators and create_POA() Nature: Clarification Summary: There is a race condition in the POA when an AdapterActivator is invoked as a result of a call to find_POA() or an over-the-wire request, and simultaneously another thread calls create_POA() attempting to create the same POA manually. Resolution: Accepted for Corba RTF 2.3 Revision: Section 9.3.3, Page 9-20. After first paragraph in page beginning with "if unknown_adapter raises..." add the following Note: Note - It is possible for another thread to create the same POA the AdapterActivator is being asked to create if AdapterActivators are used in conjunction with other threads calling create_POA with the same POA name. Applications should be prepared to deal with failures from either the manual or automatic (AdapterActivator) POA creation request. There can be no guarantee of the order of such calls. Return-Path: Sender: "Jon Goldberg" Date: Wed, 08 Jul 1998 10:56:26 -0700 From: Jon Goldberg To: Jonathan Biggar CC: port-rtf@omg.org Subject: Re: Proposed resolutions for Issues 1407-1410, 1428 References: <359AE9FF.441D15F7@inprise.com> <35A3267A.181DCAD8@inprise.com> <35A38C82.5C41F600@floorboard.com> Jonathan Biggar wrote: > > Jon Goldberg wrote: > > > > Hi Folks- > > > > I've modified and am now resubmitting George's original proposal > for > > issues 1408-9, 1428 to take into account the conference call > feedback > > from July 7 (tuesday). Based on my notes, 1428 actually was not > > changed from the conference call but I've included it in this > message > > since they are all inter-dependent. My assessment of the changes > > is: > > > > 1. description of destroy changed to explicitly talk about setting > > the POA and its descendents as if they have already been destroyed > > prior to etherealization beginning for any active objects of any > > of those POAs. This ensures that all of the POAs pending > destruction > > will reject/queue (see #3 below) requests before any > etherealization > > is started. In the initial proposal, each child POA was > independently > > destroyed before destruction of the parent POA. This allows a > > possibility of infinite looping in the destruction if > etherealization > > for two child POAs cause each other to be re-created on each > > POA destruction. > > > > 2. the current description compares the POA's temporary behavior > as if > > it is in the "holding state". We agreed to change this > terminology to > > describe it as if the POA's POAManager(s) are in a state, since > POAs > > themselves do not have state. > > > > 3. We also concluded that the state in question should be > "inactive", > > not "holding". The implication of this change is that for the > > duration of the destruction (which may include etherealization), > any > > requests for that POA will now result in an exception. Once the > > destruction is complete, any new requests might cause the POA to > be > > recreated. Although this avoids deadlocks where etherealization > > attempts to access another POA which is pending destruction, it > means > > clients will see inconsistent results depending on when > destruction > > occurs for a POA. If the state is "holding", these requests will > be > > queued and the client will always see a consistent result, > although > > the request may take longer to complete depending on when POA > > destruction occurs. In addition, with change #1 above, if the > state is > > "holding" deadlocks can occur if one POA's etherealization depends > on > > another POA being destructed. I'm stating this implication here > to > > make sure that the RTF voters are all aware of the issue. > > I don't like this much because it does cause visibility of the > destroy > process to the clients, particularly if the destroy comes from a > call to > ORB::shutdown(). However, I don't see a good way around this. > Perhaps > it would be a good idea to also state that when ORB::shutdown() is > called that the input side of the protocols (GIOP) are blocked > before > the POAs are destroyed in order to avoid client interaction while > the > shutdown is processing? Then the process would look like this: > > 1. Someone calls ORB::shutdown(). > > 2. All server side protocol connections are blocked from processing > further input. > > 3. All pending requests are drained from the ORB. > > 4. Once the last outstanding reply is sent on a server connection, > the > connection is shutdown (via CloseConnection if GIOP). > > 4. All POAs are destroyed. In the case of ORB::shutdown this may work because all new requests will be rejected and therefore POAs can be destroyed without deadlock or infinite looping. If the ORB isn't being shutdown, however (we are just destroying a POA and its descendants), new requests can still come in and we're back in the same hole as before. > 5. Any other ORB level cleanup is done. > > Of course this could be left as a quality of ORB implementation > issue, > but that is how I would implement this. After further consideration, I no longer think it is tolerable for the POAs to behave as if INACTIVE. The common case is POA destruction without mutual reference to sibling POAs, and this should not be penalized by opening up a window for client requests to be rejected. Furthermore, I no longer think it is possible to solve this deadlock problem. I think the best we can do is as follows: 1. use the scheme outlined in my revised proposal which first marks the POA and its descendants as pending destruction. 2. keep the original meaning of 'pending destruction', which means they behave as if their POAManagers are in the HOLDING state. The POAs will not behave as if their POAManagers are in the INACTIVE state because this surfaces the destruction to clients which is the wrong behavior. 3. declare that POA etherealization which depends on access to other POAs that are pending destruction is dangerous and should be avoided since it can cause deadlocks. I think #1 and #2 are necessary to maintain atomicity (a POA should not be recreated if its parent POA is pending destruction) and to protect clients. any thoughts? -Jon Return-Path: Date: Thu, 09 Jul 1998 12:40:46 -0400 From: Paul H Kyzivat Organization: NobleNet To: Jon Goldberg CC: Jonathan Biggar , port-rtf@omg.org Subject: Re: Proposed resolutions for Issues 1407-1410, 1428 References: <359AE9FF.441D15F7@inprise.com> <35A3267A.181DCAD8@inprise.com> <35A38C82.5C41F600@floorboard.com> <35A3B2CA.694FC370@inprise.com> Jon Goldberg wrote: > After further consideration, I no longer think it is tolerable for > the POAs to behave as if INACTIVE. The common case is POA > destruction > without mutual reference to sibling POAs, and this should not be > penalized by opening up a window for client requests to be rejected. I agree. > > Furthermore, I no longer think it is possible to solve this deadlock > problem. I think the best we can do is as follows: > > 1. use the scheme outlined in my revised proposal which first marks > the POA and its descendants as pending destruction. > > 2. keep the original meaning of 'pending destruction', which means > they > behave as if their POAManagers are in the HOLDING state. The > POAs will not behave as if their POAManagers are in the INACTIVE > state > because this surfaces the destruction to clients which is the > wrong behavior. > > 3. declare that POA etherealization which depends on access to other > POAs that are pending destruction is dangerous and should be avoided > since it can cause deadlocks. I am not sure I fully understand how you want this to work, but I think there is a problem: It should be possible to destroy all the POAs by destroying the root POA, and have something reasonable happen. This is implicitly what happens during shutdown, and it ought to be possible to do the same thing explicitly. But doing something reasonable in this case means that etherealization ought to take place (except perhaps in pathological cases). If destroying a parent POA first marks its entire tree as destroyed, and then starts etherealizing things, the POAs at the bottom of the tree can't use servant managers that are nearer to the root. The best way I can see to make this work reasonably is to have destroy on a POA first work bottom-up, destroying the most derived children first while leaving the higher up POAs free to service calls to any servant managers they may contain. Once a POA has given its children a chance this way, it can finally make itself unusable and etherealize its own children. This process eventually percolates all the way to the the POA that the first destroy request was sent to. There are still failure modes - when a POA has a ServantActivator in a peer or child POA, or an etherealize method that activates new objects in another POA. In the worst cases etherealization of some servants fails. I would just like to get the common cases working reasonably. > > I think #1 and #2 are necessary to maintain atomicity (a POA should > not be recreated if its parent POA is pending destruction) and > to protect clients. > > any thoughts? Yes - above.