Evolutionary approach for combinatorial testing of software product lines
Software Product Line (SPL) focuses on common features reusability, formulated for different software products. Complete testing on the entire SPL is known to be unfeasible. This is due to the very large number of possible products to be produced; configured using all possible sets of features i...
Saved in:
| Main Author: | |
|---|---|
| Format: | Thesis |
| Language: | English |
| Published: |
2020
|
| Subjects: | |
| Online Access: | http://psasir.upm.edu.my/id/eprint/91051/1/FSKTM%202020%2014%20IR.pdf http://psasir.upm.edu.my/id/eprint/91051/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Summary: | Software Product Line (SPL) focuses on common features reusability,
formulated for different software products. Complete testing on the entire
SPL is known to be unfeasible. This is due to the very large number
of possible products to be produced; configured using all possible sets
of features in the SPL.
Combinatorial Testing (CT) has been widely used in software testing as
it is able to generate test input for a single software product that deviates
from exhaustive testing, nevertheless proven to be effective. In SPL
testing, to generate minimal test configuration that maximizes t-wise
coverage is not trivial, especially when dealing with a huge number of
features and when constraints have to be satisfied, which is the case in
most SPL systems. Two salient obstacles in SPL testing are identified;
(1) time required to cover feature configuration testing for large scale
Feature Models and higher strength of t-wise testing, and (2) insufficient
coverage of feature interaction towards higher fault detection due
to uniform strength of feature combination. In the case of unfeasible
higher strength CT-based testing for SPL, this thesis proposed a relaxed
version of Covering Array (CA), i.e. Variable-strength CA (VCA). In
highly configurable systems, the entire configuration spaces contain
several groups of sub-configurations that have different level of risks
that can cause failures. By focusing on these group of subconfigurations,
the space of testing artefacts can be reduced, while still
maintain the results of testing. Practical Combinatorial Testing is challenging, such that it is hard to
decide what level of interaction strength to be applied to which different
groups of features. There are two inherent challenges; (1) to decide
what value of interaction strength one should apply, and (2) to choose
which group or set of features the interaction strength should be applied
to. For these reasons, part of this thesis presents and evaluates a
new technique aimed at tackling all the mentioned difficulties in the
combinatorial testing of SPL system. The approaches are (1) adopting
an Estimation of Distribution Algorithm approach to aid the construction
of a covering array driven by second order feature dependence,
and (2) construct a feature configuration dependence graph to assist in
building the variable-strength covering array. These techniques are
implemented as COTED (Combinatorial Testing using Bivariate EDA).
Experiments have been done to gauge the performance of COTED from
three different facets; (1) the ability to reduce test configuration redundancy,
(2) the ability to achieve better rate of interaction coverage and
(3) the ability to detect high number of faults. Four empirical studies
conducted to compare COTED against other approaches (namely ICPL,
LOOKUP and CASA). The statistical analysis of the results obtained
from the experiments are promising. For the first facet, COTED is comparable
with LOOKUP, but a slightly better redundancy is achieved by
ICPL with significant level of 0.01. For the second facet, COTED able to
outperform ICPL and LOOKUP with significance level of 0.1. For the
third facet, the two sets of VCA test configurations (i.e. (i) 2-wise and
partial 3-wise, and (ii) 2-wise, partial 3 and partial 4-wise) generated by
COTED are able to achieve competitive fault detection rate (with significance
level of 0.05) using less number of test configurations. The result
shows that COTED increase the interaction coverage rate, and
managed to produce less number of test configurations to detect an
almost same number of faults as the other strategies, hence, suggests
that it is able to improve Combinatorial Interaction Testing (CIT) for
SPL. |
|---|