About Scalable Test Design

A scalable test design makes it possible to use combinatorial functions like Covering Arrays, to postpone choices of what to test and it enables experimentation and improves maintainability of the test.

How it works

With Parameterized Validation it becomes possible to specify the choice of what to test and to specify how to validate results, but the two specifications are intimately related and must match each other in the relation:

{ subset of choice of test data } * [ specific validation ]

With a scalable test design, the subset of the chosen factor combinations is determined by matching:

{ full choice of test data | matching set } * [ specific validation ]

The matching set is the set of partial factor combinations that determines when a specific validation applies. For example, for a CRUD system, expected behavior may be given from the state of the involved objects but unrelated to other descriptive factors. This pattern can be nested to support validations involving multiple independent factors (e.g. because several independent validations must be configured).

With a scalable test design, the choice of what to test is maintained in one place, while the matching sets and mappings onto appropriate validations are maintained separately.

The benefit of this separation is that the choice of what to test can be adjusted dynamically without worrying about validation: For example, changing a test from using a pair-wise strategy to a strength-3 covering array including adapting all validations and flows, can literally be achieved in seconds.

QS supports scalable test designs through a general expression language with support for wildcards, combinatorial operators, grouping, functions and matching. These mechanisms enable very concise definitions of matching sets and mappings.

Comparing the matching approach of determining applicable validation to the use of programmed logic (see also parameterized validation) they achieve almost the same thing, but with important differences:

  • If a matching set is incomplete then unmatched factor combinations will be discarded (this is detectable by counting resulting tests). Incomplete programmed logic leads to undefined behavior and is not  easily detectable.
  • If a matching set contains an error then factor combinations will be mapped to the wrong validation and a test will fail. Errors in programmed logic lead to undefined behavior
  • It is always easy to determine what a matching set matches as the full set can be viewed directly. Determining the behavior of programmed logic depends on the complexity of that logic and may not be simple.