Towards Model-Driven Test Case Concretization for End-to-end Combinatorial Testing

MoDeVVa 2024, Linz, Austria

Léna Bamouh

Nantes Université

Erwan Bousse

Nantes Université

Context: Combinatorial Testing

Context: Combinatorial Testing

Goal

Efficiently identifying relevant combinations of input arguments for a given System Under Test (SUT), and use combinations to design test cases.

How?

Define a combinatorial model, i.e. an abstraction of the input domain, and use it to generate a set of abstract test cases (ATCs) satisfying a coverage criterion.

intro combinatorial testing.drawio

Example: combinatorial testing of isPalindrome (1)

Method signature and example calls

/**
 * Check whether a word is a palindrome.
 *
 * @param word The word to check.
 * @return true if word is a palindrome, false otherwise.
 * @throws IllegalArgumentException if word is either null, empty,
 *                                  contains a special character,
 *                                  or contains a digit
**/
public static boolean isPalindrome(String word) throws IllegalArgumentException
Example calls:

  • isPalindrome("anna") returns true,

  • isPalindrome("john") returns false,

  • isPalindrome("hello!") throws an IllegalArgumentException.

Example: combinatorial testing of isPalindrome (2)

Definition of a combinatorial model

  • Use of Microsoft’s PICT language to define a combinatorial model

  • 5 abstract parameters to cover relevant word contents (‘_’ denotes "no value")

  • 2 constraints to forbid illegal combinations:

    • if word is null, it cannot be empty,

    • if word is empty, it cannot contain characters nor be a palindrome.

isPalindrome pict 1

Example: combinatorial testing of isPalindrome (3)

Generation of abstract test cases

  • Run pict to generate combinations, choosing a pairwise coverage criterion,

  • 6 abstract test cases obtained as a result :

isPalindrome pict 1 output

Example: combinatorial testing of isPalindrome (4)

Manual definition of concrete test cases

  • A Concrete Test Case (CTC) must be manually defined for each ATC :

    • a valid concrete value must be defined for each input parameter (here, word)

    • an oracle must be defined, based on the SUT specification

isPalindrome atc5.drawio

↓

isPalindrome ctc5.drawio

Example: combinatorial testing of isPalindrome (5)

Manual implementation of a test script

A test script must be manually implemented for all CTCs.

isPalindrome ctc5.drawio

↓

@Test
void testIsPalindromeCTC5() {
    assertThrows(IllegalArgumentException.class, () -> isPalindrome("!anna!"));
}

The missing piece: automated concretization of ATCs

  • Manual work still required for CTCs definition (including input data selection and oracle definition) and test script implementation

  • Can be significant with important combinatorial complexity (ie. many combinations)

Proposal

A first end-to-end model-driven approach to automatize the concretization of abstract test cases (ATCs).

Contributions

Approach overview

overview.drawio

Step 1: PICT combinatorial model (1)

overview 1.drawio

Step 1: PICT combinatorial model (2)

Oracle-enhanced PICT combinatorial model

Problem

Oracles must be manually defined when defining CTCs

Proposal

  • Extend the combinatorial model with an Oracle variable that can be:

    • return<Java expression> (check the returned value)

    • throws<Java exception type> (check that an exception is thrown)

    • undefined (means the oracle must be manually defined)

  • Add constraints specifying what oracles should be generated in combinations

Step 1: PICT combinatorial model (3)

Example with isPalindrome − oracle definition in the PICT model

isPalindrome pict oracles

Cannot (yet) express oracles that depend on input parameters

Step 1: PICT combinatorial model (4)

Example with isPalindrome − resulting (oracle-enhanced) ATCs

isPalindrome atc.drawio

Step 2 and 3: ATC model generation (1)

overview 2.drawio

Step 2 and 3: ATC model generation (2)

Combinatorial metamodel and ATC metamodel

mm1.drawio

Step 4: Concretization into a CTC model (1)

overview 3.drawio

Step 4: Concretization into a CTC model (2)

Data generator

Problem

Input data must be manually defined based on ATCs

Proposal

Implement a data generator that must comply with the following criteria:

  • Must be a public and static Java method,

  • Must have one String parameter per parameter of the combinatorial model,

  • The return type must match the input parameters of the tested Java method (in a List is multiple input parameters).

Step 4: Concretization into a CTC model (3)

Example with isPalindrome − constraint programming (CP) data generator

isPalindrome data generator signature:

public class PalindromeDataGenerator {
    public static String generateData(String nullWord,
            String emptyWord, String specialCharWord,
            String digitCharWord, String generatePalindrome) { … }
}

isPalindrome data generator helper method (CP-based using Choco):

private static void addPalindromeConstraint(Model model, IntVar[] word) {
    for (int i = 0; i < (word.length + 1) / 2; i++)
        model.arithm(word[i], "=", word[word.length - 1 - i]).post();
}

Step 4: Concretization into a CTC model (4)

CTC metamodel

mm2.drawio

Step 4: Concretization into a CTC model (5)

Example with isPalindrome − resulting CTC model

isPalindrome ctc

Step 5: Test script code generation (1)

overview 4.drawio

Step 5: Test script code generation (2)

Example with isPalindrome − resulting test script snippets

Java code generated for CTC3

@Test
void testIsPalindromeCTC3() throws IllegalArgumentException {
    actualValue = isPalindrome("baaaa");
    assertEquals(false, actualValue);
}

Java code generated for CTC4

@Test
void testIsPalindromeCTC4() {
    assertThrows(IllegalArgumentException.class,
        () -> isPalindrome("a0a0a"));
}

Tool support: LNS-TestCrafter

  • Prototype implemented as a Java annotation processed by a Maven plugin,

  • Test scripts are automatically generated when compiling the project with Maven.

Example of GenerateTest annotation used on isPalindrome

@GenerateTest(
        PICTInputName = "palindrome_model.pict",
        generatorPackage = "data.generation.PalindromeDataGenerator",
        generatorMethod = "generateData")
public static boolean isPalindrome(String word) throws IllegalArgumentException { … }

Evaluation and Conclusion

Evaluation (1)

Research questions and experimental setup

Research questions (RQs)

  • How does the combined effort of implementing a data generator and extending the combinatorial model with an oracle definition compare to the effort of implementing test scripts manually?

  • How does this comparison vary with combinatorial complexity, e.g., when changing the target coverage criterion?

Experimental setup

  • 5 example methods from the field of software testing (isPalindrome, getTriangleType, getDaysInMonth, findCommand, validatePasswordStrength)

  • Effort measured in lines of code (LoC)

Evaluation (2)

Resulting LoC measured on the 5 case studies

Graphic eval

  • isPalindrome, getTriangleType, and getDaysInMonth are not "breaking even" (ie. more effort required than coding manually),

  • findCommand "breaks even" with 3-wise,

  • validatePasswordStrength "breaks even" with both pairwise and 3-wise criteria.

Takeaway

Approach "worth it" when the combinatorial complexity is high enough.

Conclusion

Problem

Combinatorial testing requires manual work in its last stages (ie. concretization)

Contribution

A first end-to-end model-driven approach to automatize concretization with data generation (eg. using constraint programming)

Possible research directions

  • More complex input data (eg. nested objects)

  • More complex oracles in the combinatorial models

  • Other types of data generators (eg. AI-based)

Thank you for your attention!

https://gitlab.univ-nantes.fr/lns-testcrafter