Issues for UML Testing Profile Finalization Task Force

List of issues (green=resolved, yellow=pending Board vote, red=unresolved)

List options: All ; Open Issues only; or Closed Issues only

Issue 6290: Details of the Standalone Model
Issue 6291: XMI Schema (Profile and Standalone Model)
Issue 6292: Test Suite / Test Case
Issue 6293: Verdict in Test Case (Standalone Model)
Issue 6294: Commonalities between test suite and test case (standalone model)
Issue 6295: Traces (Standalone Model)
Issue 6296: Traces for test cases and test suite (Profile)
Issue 6297: Traces vs. Log/Journal/… (Profile and Standalone Model)
Issue 6298: Data Pools and Data Partitions (Profile and Standalone Model)
Issue 6299: Data Picker/Data Selection (Profile and Standalone Model)
Issue 6300: Simplification of the Standalone Model
Issue 6301: Relation of Test Suite and Arbiter (Profile)
Issue 6302: Load Tests (Profile)
Issue 6303: Activity Diagrams (Profile)
Issue 6304: Issues with the Load Testing Example
Issue 6305: Editorial for Fig. 4
Issue 6306: Arbiter Semantics
Issue 6307: Synchronization/Coordination of Test Components
Issue 6951: Relationship between Arbiter and Behavior (Standalone Model)
Issue 6952: Default refining from Behavior (Standalone Model)
Issue 6953: Test Case execution in a Suite or Test Case Context
Issue 6954: Constrained semantics for UML constructs
Issue 7104: Fig. 28 and Fig. 29 use a syntax that have not been defined by U2TP
Issue 7193: Reference to TTCN-3
Issue 7194: Hybrid Defaults
Issue 7218: Editorial comments

Issue 6290: Details of the Standalone Model (uml-testing-profile-ftf)

The standalone metamodel is not detailed enough to enable the implementation of testing profile compliant tools. In particular, many of the classes are having no attributes. Hence, a behavioural and further semantic foundation should be added to the standalone metamodel

The UML Testing Profile lacks currently a definition of XMI schema for the profile and the standalone model. These are needed to allow tool interchange of test specifications. Here, the reference to an XML DTD in the compliance point definition should be changed to a reference to the schema definition

Users of the testing profile are confused with the containment relation of test cases to test suites. Test cases are considered to be independent of test suites (they are designed in the test planning phase), while test suites just denote one possible execution of test cases.

Verdicts are part of a test case specification; however, a test case specification may contain several different verdicts - as different test sequences (all part of that test case) may lead to different verdicts. Hence, there is not a single test verdict and the attribute verdict of a test case should be deleted.

Summary:
Test suites and test cases share certain characteristics, i.e. a behavior, test objective, and trace. Hence, a common superclass should be introduced

Traces are currently related to test cases only, but a trace should be possible for a test suite as well.

Summary:
It should be possible to have traces for test suites and test cases. These traces should be defined comparable to the standalone metamodel traces. 

The notion trace could be confusing as it is used e.g. in UML 2.0 interactions to define the interaction semantics. Hence, a renaming is proposed.

The notions of data pool and data partition are used throughout the document but explained only in the terminology section. In the profile, it is argued that data pools and data partitions are modeled by use of standard UML 2.0 concepts, but it is not explained how. In the standalone model, both are missing completely. 
Hence, data pool and data partition should be added to the standalone model. In addition, a standard way of representing data pools and data partitions should be defined for the profile (by defining stereotypes for data pools and data partitions).  

In testing, typically test cases are executed with different test data. For that, testdata is selected along different selection strategies. However, currently test data selection is not a separate concept in the UML testing profile. 
The central point is the separation of test data specification and test selection for test execution. An additional interface to plug in different test selection strategies could be a solution.  

There are certain concepts in the standalone model which are of no use to the user: the default application, validation action and log action belong to behavior, which is not described in the standalone model. Hence, a tool vendor cannot really handle these concepts, e.g. a user could not really define a validation action outside of a context of a concrete behavioral specification.
Another view on this: we need for a tool vendor a compliance level which is an intermediate step from their existing tools to the standalone model. This compliance level should exclude the behavior definition, i.e. it should be open to any kind of test behaviors. 

The definition for a test suite in the profile has a condition: "A test suite must contain exactly one property realizing the Arbiter interface." This gives a constraint on the metamodel. The property is not named.

The definition of load tests is cumbersome. There are no high-level operators to enable the parallel and quasi-simultaneous execution of test components - following a certain distribution function and realizing different test behaviors. Currently, one has to define e.g. a separate interaction to spawn the test components like in the example given in the U2TP specification.

The current specification of the UML Testing Profile does not address the specification of test behaviors with activity diagrams, although users see a need to have also activity diagrams been supported by the testing profile.

The arbitration interaction referenced in Fig. 36 is not shown. 
The diagram in Fig. 39 defines a state machines and not an interaction. Hence, the sd in the header should be deleted. 

There is an "hanging" e top left of Fig. 4. - simply delete it.

The current specification says nothing about the semantics of an arbiter for example how the arbiter gets to know that a test case finished and that the arbiter has to calculate the final verdict.  

The current specification does not offer high-level concepts for functional synchronization and coordination between test components like rendezvous, joint start and joint termination

The current standalone model does not allow to define a behavior for an arbiter

In the standalone model, default inherits from behavior, which would allow to define a test case or a test suite as a default.

In some contexts where a Test Case (B) is reused (i.e. invoked) from another Test Case (A), there is an ambiguity to the actual intent of the tester
“A” needs to assess that “B” passes to set its own verdict
“A” is reusing the behavior of “B”. The verdict set for “B” by the arbiter is not relevant for either “A” or “B”
Proposal: A Test Case has a mandatory argument, which decides if its verdict should be logged in the Test Log. If the boolean is true, the verdict is logged, otherwise it is not

Introduction of an ordered alt operator (“if-then-else”) operator. Mapping to more complex expression

the special comment declares a default - its behavior is given in a
behavioral diagram.


Hence the right rectangular box in Fig. 28 is superfluous. Also, the
definition of the statemachine for the default (in Fig. 28 and Fig. 29)
should simply contain the default identifier - i.e. hweDefault - and not
<<default>> statemachine hweDefault - both <<default>> and statemachine
are superfluous?!



Furthermore: the use of statemachine within the header of a
statemachine's behavioral diagram is repeated - i.e. in Fig. 30 - I
could not find in the superstructure document that this would be allowed

Technical references to TTCN-3 should not be part of the U2TP specification

Summary:
Hybrid defaults (i.e. defaults in state machines for test cases defined in interactions) cause the problem of defining the semantics for a hybrid behavior definition. In particular, the interworking of diagrams of different types is in general is not well defined in UML 2.0.
Hence, hybrid defaults should be removed from the U2TP specification

In the editing process for U2TP several editorial comments to be solved by the FTF have been made. These have to be addressed