Issue 6955: Data guards on observations Jira Issue UTP11-1
Issue 6956: Grey box testing Jira Issue UTP11-2
Issue 15726: OCL expressions should be used for precise description of constraints Jira Issue UTP11-3
Issue 15914: Test scheduling should be supported explecitely Jira Issue UTP11-4
Issue 15927: Test Cases as an Operation Jira Issue UTP11-5
Issue 15933: term data value issue Jira Issue UTP11-6
Issue 15934: Test Context Jira Issue UTP11-7
Issue 15935: Test Case Types & Instances Jira Issue UTP11-8
Issue 15936: Anatomy of a Test Case Jira Issue UTP11-9
Issue 15937: Visual & Standalone Test Cases Jira Issue UTP11-10
Issue 15938: Associations among Test Cases Jira Issue UTP11-11
Issue 15940: Test Maps Jira Issue UTP11-12
Issue 15943: Default Jira Issue UTP11-13
Issue 15944: getDistributionInterval Jira Issue UTP11-14
Issue 15998: Alter predefined interfaces to stereotypes with constraints Jira Issue UTP11-15
Issue 15999: Align test-related actions for Interactions with UML 2.4 Jira Issue UTP11-16
Issue 16568: Invalid Figure B.14 Jira Issue UTP11-17
Issue 16899: Attribute CodingRule::coding should be typed by ValueSpecification Jira Issue UTP11-18
Issue 16901: TestComponent stereotype should extend Property Jira Issue UTP11-19
Issue 16905: Constraints on superclasses needs to be overwritable for test data specifications Jira Issue UTP11-20
Issue 17228: Verdict should be renamed to VerdictKind Jira Issue UTP11-21
Issue 17230: Figure B.28 - Transaction detail - is erroneous Jira Issue UTP11-22
Issue 19018: Unnecessary Comment in the normative XMI representation of UTP Jira Issue UTP11-23
Issue 19084: Issue in the UTP 1.2 normative XMI file Jira Issue UTP11-24
Issue 6955: Data guards on observations (uml-testing-profile-rtf)
Click here for this issue's archive.
Source: Motorola (Mr. Paul Baker, nobody)
Nature: Uncategorized Issue
Severity:
Summary:
For each operand the guard has to be on each leading event within an alternative
Deferred due to time restrictions.
Monitoring of interfaces between components within the SUT.
Since UTP 1.1 rather addresses technical and conceptual issues, the elaboration of such a new concept is deferred to one of the next revisions. Disposition: Deferred
Rational ##################### OCL is the OMG�s formal, MOF-based language to express fine-grained constraints in MOF-based models. OCL expressions are unambiguous, precise and computable. Issue ##################### All constraints in UTP specification are defined with natural language. In order to help readers (tool vendors, tester, modeler) to implement, understand and apply the stereotypes of UTP properly, it would be helpful to use formal OCL expressions for these constraints. The natural language description may remain in the specification, meaning the OCL supplements the already existing description. The benefits would be two-folded: Both readers with and without OCL knowledge would be able to understand the specification. UML 2.3 Superstructure, section 18.3.2 Extension, subsection Semantics explains how OCL constraints can be exploited to define restriction on the extended metaclass.
Deferred due to time restrictions
Some test cases might required additional hardware that is only available in specific timeframes. This means the test cases must be scheduled in detail and in certain sequences. Also, test cases may depend on each other. Corresponding model element assignments can be established in principle with the current UTP specification, but a corresponding rule-set or a basic standard is missing. Such kind of rule-set standard is necessary for potential tooling support of scheduling aspects coming out of an UTP test model.
Deferred due to time restrictions
Source: Lockheed Martin, Jon Hagar Specification: UML Testing Profile Section: fig 6.2 Nature: Issue Severity: med Summary: Test Cases as an Operation There isn�t a formal diagram notation for an operation in the UTP; therefore you can�t show it on a diagram using a standard notation. Tool vendors have provided solutions but they are not part of the standard. Figure 6.2 shows test case and operation (metaclass) Resolution: Provide notation definition of operation to be used throughout the standard. Revised Text:
Deferred due to time restrictions
Source: Lockheed Martin, Jon Hagar Specification: UML Testing Profile Section: term data value Nature: Issue Severity: med Summary: The term data value is used many places and seems to be a root for data definitions, e.g. pools, but is not semantically defined. Resolution: Create a semantic definition for data values
Deferred due to time restrictions
Today, �TestContext� is a stereotype of a "StructuredClassifier". However, test contexts must be able to contain anything that is useful to model test specifications, including other test contexts, test cases, test components, and even different kinds of diagrams.
Acutally, the text context was introduced to represent something that provides the structural foundation for the test cases, such as test components, utility componentes, the sut and so forth. Additionally, a test context is by definition something that is instantiable in order to dynamically invoke test cases as operation, to parametrize them with data values (in terms of instance specification). The way as a test context is described in the issue rather refers to something like a dedicated �TestPackage�, a packaging mechanism that combines any test relevant information. It must be analyzed whether there are some basic information that would justify the existence of a dedicated test package. Technically, it would definitely make sense for tool developers to have dedicated concept being present in the profile. From the current point of discussion, this issue will be deferred to a further revision. Disposition: Deferred
A single test case is a type that needs to be instantiable (i.e. executed) many times. Instances of a single test case may either be executed in a sequence or in parallel (in which case multiple instances of the same test case exist at the same time). When a single instance of a test case has been executed it should mainly be frozen but may be retained for historical reasons.
This issue will be merged with issue 15927 for resolution
A single test case must be stateful as it controls the sequence of the individual test steps and may need to store temporary data while being executed. These requirements make it difficult to have test cases based on operations. On the other hand, test cases need to be parameterized to achieve some level of genericity, which is less common to behaviors.
This issue will be merged with issue 15927 for resolution
A common early step in specifying test cases is identifying a set of test cases. Today, �TestCase� is a stereotype of an "Operation" or a "Behavior" which is a specialization of a "Class". To support test case identification a test case should be a first-class model element that may exist standalone and that may visually be recognized as a test case.
This issue will be merged with issue 15927 for resolution
Test cases must support various associations among them so that they may be organized into families of related test cases.
This issue will be merged with issue 15927 for resolution
When identifying test cases one of the most important goals is to minimize the number of test cases while maximizing the test coverage. The core approach to achieve this is to employ various mechanism that facilitate reusability. Therefore, it is very helpful to visualize families of related test cases and their associations on dedicated diagrams.
A diagram for associating test cases in an early phase of a test process makes definitely sense and should be considered for one of the next minor revisions. Since test cases can be expressed as UML::Behavior, and UML::Behavior is a subclass of UML::Class, the most appropriate diagram would be a class diagram with additional symbols. In order to achieve this, it must be clearly and precisely states what concepts ans relationships are needed among test cases. Further more, definitions of monitors for test cases should be included; a test map may define behavioral description stating what should happen, if a monitored situation for a currently execute test case happens (write a log somewhere or interrupt the test case execution). The definition of such �behavioral� monitors must be aligned with issue 6956. Disposition: Deferred
The wildcard * to indicate any kind of default is particularly on state machines uncommon and inflexible.
This issue is rather related to the non-defined concepts of any message (Fig. 6.30), any state (Fig. 6.31) and any event (Fig. 6.32). Those concepts are intuitively used in the UTP example sections, but neither part of the UML nor of the UTP. The wildcards, introduced by UTP, are dedicated to values of test data instance specifications (for slot values). In the example sections, the UTP uses the wildcard notation for defaults to capture various situations: undefined messages are received (Fig. 6.31) a set of �exception handling� transitions (Fig. 6.32) are intended to augment each state in the behavior referencing the default, in a way that all transitions of the generic state (state with * as its name) are copied to each state in the test case state machine an undefinied event is recognized in the state machines event queue (Fig. 6.33). No doubt, those are very powerful and helpful concepts, however, they are not defined in the semantics sections of the UTP. Further minor revision must address this issue in order to produce a more precise UTP specification. Disposition: Deferred
The role of operation "getDistributionInterval" on class "DataPool" (in figures 6.39 and 6.41) is not clear.
Deferred due to time restrictions.
The UTP predefines three interfaces, that are Scheduler, Arbiter and Timer. Mixing interfaces and stereotypes, which are supposed to work with those interfaces, is confusing and complicates the implementation of the profile. The idea of those interfaces is straight forward, however, the technical realization is hard to achieve due to mixed meta layers. UML::Interfaces instances are located at meta layer M1 (model layer), whereas Stereotype instances are located at meta layer M2 (meta-model layer). Since the profile mechanism allows the definition of constraints for the base class of a stereotype, it is possible to express the semantics of the predefined interfaces with stereotypes and constraints, being applied on them. Even if a UML profile tool does not include OCL evaluation, the profile would be still easily implementable. The key point is here: Including OCL constraints into the semantics of a stereotype does not restrict the technical feasibility of the profile. Rather, the modeler would be responsible to respect the constraints of the stereotype. If a tool makes use of OCL expressions, the validity of the model would be checked automatically.
Deferred due to time restrictions
In the current UTP specification, the different test actions (StartTimerAction, ValidationAction, etc.) are indirectly bound to a lifeline by being referenced from an ActionExecutionSpecification that covers a lifeline, representing a structured classifier with <<TestComponent>> stereotype applied. This was necessary because there was no adequate meta model element in Interactions for placing a single action onto a lifeline. With UML 2.4, the meta model of Interactions have been reworked significantly in the way that the meta model element OccurrenceSpecification became a concrete class, specializing InteractionFragment. An OccurrenceSpecification has a reference to exactly one lifeline it covers. This the precise redefinition of test actions for Interactions, using the new UML 2.4 semantics. Therefore, each test action (StartTimerAction, StopTimerAction, ValidationAction, etc.) shall extend UML::OccurrenceSpecification directly. Since OccurrenceSpecifications do not have a notation a priori, each test action has to define a dedicated notation (for some actions, there is already a notation defined, e.g. TimeOutMessage). This makes the actions easily and freely positionable on the lifelines. By doing so, the applicability of UTP would be significantly simplified. Besides, the memory footprint in the background would be extremly decreased, due to the fact that the event would be bound directly to the OccurrenceSpecification instead of creating a lot of elements (ActionExecutionSpecification, Action, Event, Input/OutputPins...)
Deferred due to time restrictions
Given by constraint 6 of UML::Message (Section 14.3.18 in UML 2.4 Superstructure), messages are not allowed to cross boundaries of a CombinedFragment. Gates manage the connection of messages outside of a CombinedFragment to messages inside the CombinedFragment. So, in general, a cfragmentGate should be used in this Figure. However, in picture B.14 a Default is depicted. Defaults express behavior for exceptional situation within a test case specfication, so actually, the behavior of a Default is added to the receiving test component lifeline for a particular message (shown in Figure B.13), or for a set of messages (not mentioned in the spec at all). Thus, the two boundary-crossing messages shown in B.14 do not have a real counterpart in Figure B.13 (they are just additional messages). For resolution, I suggest to use found messages (UML::Message with MessageKind::Found) within the InteractionOperands of the determAlt CombinedFragment, to indicate they are additional message for which no real counterparts exists in the interaction applying a Default.
Deferred due to time restrictions. Disposition: Deferred
The specification states that coding rules should be defined outside of the utp model and just being referred from the utp model. In some situations, it can might be required to include a precise coding and decoding of values already in the model. Therefore, the coding attribute should be typed by ValueSpecification, which would allow both merely naming the coding rule that should be used by an external test execution environment (LiteralString) and a formal representation of the coding/decoding mechanism itself (Expression/InstanceValue/OpaqueExpression)
In situations where components already exist which shall be reused for the test architecture, it is contra-productive that test component only extends StructuredClassifier. This requires always the creation of a completely new test component (maybe using generalization) for the test architecture. Treating TestComponent, in addition to its current representation, similar to <<SUT>> (which extends Property) would result in more flexibility since existing classifier can be reused for being both the SUT and a TestComponent in different tets context. This is in particular helpful for reusing existing classifier. This way would ot restrict the situation where completely new tets component classifiers must be created (maybe though generalization from already existing classifier in the system model). Current approaches would not become invalid, in contrast, this would result in a more precise and clear test configuration.
UML supports the substitution principle of the object-oriented paradigm among other by not allowing the violation of constraints on superclasses in specialized subclasses. This is fine for the definition of system specifications. Testing, however, has to deal with invalid situations and data types. The most common test design/test data design technique relies on that fundamental principle of identifying "invalid" partitions of data types - the equivalence classes. Invalid means that the data partition does not comply with the constraints that is either applied to an interface operation or the data type itself. In UTP, data partitions are defined as stereotypes on classifier, hence it is possible to reuse existing type definition (from an existing system model) and to create data partitions for that type (let's called it base type) that divides the base type into several different data partitions. As an example, consider the following: class A{ + i : Integer {i >= 0 and i <= 100} } Applying the equivalence class method, this base data type would result in three equivalence classes/data partitions: 1. DP_1 = i >= 0 and <= 100 //valid 2. DP_2 = i < 0 //invalid 3. DP_3 = i > 100 //invalid If we want to reuse the base data type for the design of test specifications, we would have to generalize the already existing class A by three different data partitions like: DP_1 extends A{ //valid + a : Integer {i >= 0 and i <=100} } //constraint inherited by class A DP_2 extends A{ //invalid + a : Integer {i < 0} } DP_3 extends A{ //invalid + a : Integer {i > 0} } Creating DP_2 and DP_3 would result in a contradiction, since UML disallows constraints to be deactivated or overwritten. DP_2 would look like this: DP_2 extends A{ //invalid + a : Integer {i < 0 and i >= 0 and i <=100} } This requires a mechanism to explicitly target constraints in superclasses which are overwritten (not redefined) in subclasses with <<DataParition>> applied. To summarize: Test specifications address a larger set of data types than sytem specification, since they have to stimulate the system with data which are invalid by the system specification. A possible and minimal inversive solution could the following be: Introduce a new stereotype <<OverwritingConstraint>> or <<TestConstraint>> that has at least one tag definition, namely: + overwrittenConstraints : Constraint [1..*] whereas each Constraint being referenced by the tag definition must be a constraint initially introduced by the base type of the data partition or one of its subclasses.The enumeration Verdict should be renamed to VerdictKind, to be consistent with UML and all other UML profiles.
Figure B.28 depicts an interaction between a tect component lifeline and several SUT lifeline. The first two and the last two messages invoke an operation called checkBalance with a actual parameter p.euAccount/p.usAccount, where p is of type TrxnData. The type TrxnData is specified in Figure B.23, and in this Figure (which is the only Figure for the definition of TrxnData) there are no attributes for TrxnData called euAccount or usAccount, but account.
In the normative XMI representation of the UML Testing Profile version 1.2 (https://www.omg.org/spec/UTP/20120801/utp_1.2.xmi) there is the following ownedComment element: <packagedElement xmi:type="uml:Stereotype" xmi:id="_PsR94OVOEeG84fBOY39c0g" name="TestCase"> <ownedComment xmi:type="uml:Comment" xmi:id="_ZvGtsOVOEeG84fBOY39c0g"> <body>[09.08.2012 09:16:54] ... my choice would be: Scandic Palace Tallinn, 3 VABADUSE VALJAK SQUARE, TALLINN, EE10141</body> </ownedComment> It seems to me that comment is by mistake, and not necessary in the profile definition.
I�d like to report on an issue on the normative files for the UML Testing Profile version 1.2. According to the PDF spec, section 11.4 � Test Result Analysis �, the TestLog stereotype extends the Behavior metaclass. This extension relationship is not specified in the utp_1.2.xmi file.