Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Locating Minimal Fault Interaction in Combinatorial Testing

Locating Minimal Fault Interaction in Combinatorial Testing Locating Minimal Fault Interaction in Combinatorial Testing div.banner_title_bkg div.trangle { border-color: #47132C transparent transparent transparent; opacity:0.8; /*new styles start*/ -ms-filter:"progid:DXImageTransform.Microsoft.Alpha(Opacity=80)" ;filter: alpha(opacity=80); /*new styles end*/ } div.banner_title_bkg_if div.trangle { border-color: transparent transparent #47132C transparent ; opacity:0.8; /*new styles start*/ -ms-filter:"progid:DXImageTransform.Microsoft.Alpha(Opacity=80)" ;filter: alpha(opacity=80); /*new styles end*/ } div.banner_title_bkg div.trangle { width: 317px; } #banner { background-image: url('http://images.hindawi.com/journals/ase/ase.banner.jpg'); background-position: 50% 0;} Hindawi Publishing Corporation Home Journals About Us Advances in Software Engineering About this Journal Submit a Manuscript Table of Contents Journal Menu About this Journal · Abstracting and Indexing · Aims and Scope · Article Processing Charges · Articles in Press · Author Guidelines · Bibliographic Information · Citations to this Journal · Contact Information · Editorial Board · Editorial Workflow · Free eTOC Alerts · Publication Ethics · Reviewers Acknowledgment · Submit a Manuscript · Subscription Information · Table of Contents Open Special Issues · Published Special Issues · Special Issue Guidelines Abstract Full-Text PDF Full-Text HTML Full-Text ePUB Full-Text XML Linked References How to Cite this Article Views 90 Citations 0 ePub 3 PDF 8 Advances in Software Engineering Volume 2016 (2016), Article ID 2409521, 10 pages http://dx.doi.org/10.1155/2016/2409521 Research Article Locating Minimal Fault Interaction in Combinatorial Testing Wei Zheng , Xiaoxue Wu , Desheng Hu , and Qihai Zhu College of Software & Micro-Electronics, Northwestern Polytechnical University, Xi’an 710072, China Received 14 October 2015; Revised 22 March 2016; Accepted 19 April 2016 Academic Editor: Henry Muccini Copyright © 2016 Wei Zheng et al. This is an open access article distributed under the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Abstract Combinatorial testing (CT) technique could significantly reduce testing cost and increase software system quality. By using the test suite generated by CT as input to conduct black-box testing towards a system, we are able to detect interactions that trigger the system’s faults. Given a test case, there may be only part of all its parameters relevant to the defects in system and the interaction constructed by those partial parameters is key factor of triggering fault. If we can locate those parameters accurately, this will facilitate the software diagnosing and testing process. This paper proposes a novel algorithm named complete Fault Interaction Location (comFIL) to locate those interactions that cause system’s failures and meanwhile obtains the minimal set of target interactions in test suite produced by CT. By applying this method, testers can analyze and locate the factors relevant to defects of system more precisely, thus making the process of software testing and debugging easier and more efficient. The results of our empirical study indicate that comFIL performs better compared with known fault location techniques in combinatorial testing because of its improved effectiveness and precision. 1. Introduction Combinatorial testing could significantly reduce test cost and increase quality of software system [ 1 ]. It has been proved to be effective especially in a software system where faults come from the interactions of its parameters [ 2 ]. Combinatorial testing could detect the parameter interactions that trigger the system faults rather than localize it. If a test case triggers the fault of a system, it reflects that there exists one or more defects in the program [ 3 – 5 ]. However, not all parameters in the test case are relevant to defects. If we are able to locate a parameter in the test case that is relevant to the fault, we can apply this useful information to facilitate the debugging process. In combinatorial testing, the study on fault interaction technique could be categorized into adaptive method and nonadaptive method according to the dependence between additional test cases and running results [ 6 , 7 ]. For nonadaptive methods, the generation of additional test cases does not rely on the running result of original test cases. Colbourn and McClary [ 8 ] present a nonadaptive method named Locating and Detecting Arrays (LDA). Based on basic known information such as parameters’ number, values, and faults’ number, the method applies -way Locating and Detecting Array to locate faults in software system. Martínez et al. [ 9 ] present a self-adaptive algorithm based on Errors Locating Arrays (ELA) and analyze the algorithm complexity. However, this method could only be used under the condition that the value’s number of each parameter in software is not larger than 2. On the basis of LDA and ELA, Hagar et al. [ 1 ] propose the method of Partial Covering Array (PCA) which could be used in the software with known safe value, and it presents a new combinatorial structure to generate ELA. Another category is known as adaptive method [ 10 ], whose generation of additional test cases depends on the information given by the execution of original test cases. Zeller and Hildebrandt [ 11 ] present a typical adaptive method named Delta Debugging. The main idea of this method is to identify the interaction that is relevant to the faults by modifying the input parameters. For a test case that triggers the fault, modify some of its input parameters; if the modified test case still triggers the fault, then the modified parameters are irrelevant to fault; otherwise, the modified parameters are related to fault. Based on Delta Debugging, Z. Zhang and J. Zhang [ 12 ] present a method named FIC. Similar to Delta Debugging, FIC modify one parameter in a test case with parameters once. Then repeat this process times and the minimal fault interaction is calculated afterwards. A restraint of Delta Debugging based methods is that they could only be applicable to the test case containing one minimal fault interaction, but, in real-world programs, more than one minimal fault interaction is very common. Ghandehari et al. [ 13 ] present a fault localization tool based on a failure-inducing combinations algorithm of [ 14 ]; it leverages the notion of inducing combination to locate the faults inside the source code. To locate each interaction related to faults, in this paper we present a new complete Fault Interaction Location (comFIL) method. The method includes 2 steps to locate minimal fault interactions. First, after the execution of original test cases, the test cases will be divided into 2 sets: FTS and PTS; the former contains test cases that trigger faults, while the latter contains test cases that do not trigger faults. Then we identify the set of interactions covered by FTS but not covered by PTS and name this set as candidate faulty interaction set (canFIS). Second, we generate additional test cases to select interactions in canFIS and then the minimal fault interaction set is obtained finally. 2. Preliminaries 2.1. Basic Definitions Assume there is a system under test (SUT) with input parameters as . For any , the range of parameter is denoted as . And pair is used to indicate that the value of parameter is , also denoted as ( ). For example, a model named BCSD is shown in Table 1 . The model has 4 input parameters: client browser, client operating system (client OS), server operating system (server OS), and server database. Table 1: The input model of a simple system. In this model, the range of parameter is . The input parameter model of SUT is denoted as , which means SUT has input parameters and, for each parameter , the value numbers are , respectively. Definition 1 ( -way interaction). For a set of elements , if for any , there are , then we call a -way interaction. Definition 2 (superinteraction and subinteraction). For 2 interactions, and , if , then we call the subinteraction of and the superinteraction of . An interaction is both the subinteraction and superinteraction of itself. For example, in model BSCD, interaction is the subinteraction of interaction . Definition 3 (subinteraction set). For a -way interaction , its power set is ; is denoted as the set of all subinteractions of . Definition 4 (fault interaction). For an interaction , if , then is a fault interaction. Definition 5 (minimal fault interaction). For an interaction , if for any interaction , is not fault interaction, then is minimal fault interaction. Definition 6 (additional test case function). For an interaction , additional test case function generates a test case that satisfies the following: if , then is a fault interaction; otherwise, is not a fault interaction, denoted as . Definition 7 (candidate fault interaction set). Denote as a candidate fault interaction set of . 2.2. Basic Assumptions Assumption 1. The parameters in the system are independent of each other. In many systems, there are constraint conditions between parameters. In [ 15 ] Cohen et al. study the method to generate test cases under constraint conditions. However, the constraint conditions among parameters are not the focus of fault location method, so we do not consider the dependence between parameters in this paper. Assumption 2. If an interaction causes the fault of the system, the test cases containing this interaction will trigger the fault necessarily. Assumption 3. The test cases generated by additional test case function do not contain the minimal fault interaction. For an interaction , generate a test case ; if trigger the fault, then this fault is caused by interactions in the set of subinteractions of . If contains the minimal fault interaction, the complexity of fault location will be increased. So we avoid considering it in this paper. This assumption will be explained later in the implementation of . The algorithm comFIL presented by this paper is based on the assumptions above. The algorithm is effective only under the condition that all these 3 assumptions are tenable. 2.3. Basic Inferences Inference 1. The test case that contains fault interaction will trigger the fault of system necessarily and the test case that does not trigger the fault does not contain the fault interaction. Inference 2. If an interaction is not a fault interaction, any of its subinteractions is not fault interaction. 2.4. Basic Theorems Theorem 1. For 2 interactions and , if , then . Proof. From Definition 3 (subinteraction set), we can know that and for any , , so ; namely, . Theorem 2. Divide into 2 sets: FTS and PTS. In FTS, all test cases triggered the fault of the system, ; if is the minimal fault interaction of , then . Proof. From Inference 1 we can know that, for any minimal fault interaction in , there is a failed test case that satisfies . So and ; that is, . Theorem 3. Use MFIS which denotes all minimal fault interactions set of ; for a minimal fault interaction , if and , then and . Proof. This could be directly concluded from Theorem 2 . Theorem 4. For a -way interaction , the additional test case is . In subinteraction set of , the number of interactions that do not belong to is . Proof. From Definition 4 we know that and , in which is the subinteraction of . From Theorem 1 we know that . So . 3. comFIL Algorithm 3.1. Description of comFIL Theorem 3 and Definition 5 provide a screening method for obtaining the set of all minimal fault interactions from the set of candidate fault interactions; we name the set as complete Fault Interaction Location and denote it as comFIL for short. When an interaction proves to be the fault interaction, we delete all the parent interactions of from canFIS except , while we delete all the child interactions of from canFIS except when it proves not to be the fault interaction. Under the three assumptions in Section 2 , algorithm comFIL is based on Theorems 2 and 3 and Definition 6 as theoretical core. According to Theorem 2 , we calculate the set of candidate fault interactions from the combination test cases. And, based on Theorem 3 and Definition 6 , we screen out the set of minimal fault interactions from the set of candidate fault interactions. The basic framework of comFIL algorithm is shown in Algorithm 1 . It has two input parameters and and an output parameter canFIS, which is the minimal fault interaction set. Generally the algorithm process could be divided into 2 phases. Algorithm 1: The comFIL algorithm. Generate the canFIS for fault location. Firstly, the algorithm counts the number of times that the fault interaction exists in passed or failed test cases separately (steps (2) to (12)); then the canFIS is screened out from test case set (steps (13) to (15)). Generate the minimal fault interaction set (steps (16) to (20)). Steps (17) and (18) describe the following: if a schema is a fault interaction, then we delete all its superinteraction except for in canFIS. Steps (19) and (20) describe the following: if an interaction is not a fault interaction, then we delete all elements in its subinteraction in canFIS. 3.2. Implementation of Key Functions of Algorithm comFIL (1) Implementation of subSet . The input of function subSet is an interaction , and its output is the subinteraction set of . For -way interaction , its -way subinteraction is a binary string , in which ; that is, a -way subinteraction of is , in which are not equal to each other. For example, a subinteraction of interaction is , whose binary string is . A binary string is corresponding to a certain decimal number, so, for -way interaction , its subinteraction set is the corresponding binary stings of integer set . (2) Implementation of addTF . The input of function addTF is an interaction , and its output is an additional test case . For the input , the number of its ways is not ; that is, is not a test case. From Assumption 3 we can know that ; moreover, is the only minimal fault interaction among all subinteractions of . From Theorem 4 we know that when generating addition test case , we will first ensure that there is no minimal fault interaction in subinteractions. However, if is very large and is relatively small, the numbers of interactions to be examined and additional test cases to be generated are very large. For convenience, we assume each parameter has a value, which does not associate with any fault (i.e., this value does not belong to any minimal fault interaction). This value is denoted as safe value. In comFIL, the number of times that each interaction appears in passed test cases and failed test cases needs to be recorded. For a one-way interaction, that is, a value of a parameter, the value is named the fault ratio of the value. We simply consider that the smaller the fault ratio of a parameter’s value is, the more likely this value would be the safe value of the parameter. When it comes to testing an object as we could not get the safe value of each parameter before testing, according to Theorem 4 , normally there will be an enormous number of interactions to be tested for the additional test cases generated by the interaction, resulting in the fact that the cost is very high. However, when there only exists a small amount of software defects (e.g., the software in the Delta testing phase), the number of the minimal fault interactions in the combination test cases is very small and thus there is high possibility for the existence of safe value in the parameters. The process of generating additional test cases is as follows. First, if a value of a parameter appears in interaction , the value of in additional test case is itself; otherwise it will be assigned by the value that has the smallest safe value. If many values of have the same smallest safe value, then will be assigned randomly among these values. Second, to check , if it does not belong to test suite , then is used as an additional test case; otherwise, will be regenerated. The regeneration process is to modify a parameter’s value in by assigning this parameter another value whose safe value is the smallest or second smallest, thus making sure is a subinteraction of . Then we repeat this process till does not belong to . 3.3. An Application of comFIL As shown in Algorithm 2 , foo is a method proposed in [ 13 ]. The correct prototype of foo in line (11) should be “ ”; because of the loss of “+” it becomes the wrong statement “ ,” and it is a bug of program foo. The model of foo’s input parameters is SUT (4; (2, 2, 3, 4)). The ranges of parameters are ; ; ; , respectively. If an input contains or , data stream could reach the bug of foo; meanwhile, the state of foo is different from its correct prototype. Therefore, the minimal fault interaction set is . Algorithm 2: A fault program. The program foo is tested on 2-way coverage in this paper. Test case set and test result are shown in Table 2 . The algorithm comFIL uses Table 2 as input. The first process is to screen out canFIS from Table 2 . Table 2: Test result of 2-way coverage. Table 3 shows the results of first process, in which the first column refers to the number of interactions while the first row represents the parameter of each interaction and the remaining rows represent the respective value of the parameters. Each row in Table 3 indicates an interaction from row 2 on. For example, interaction 1 is shown in row 2 (I # 1). Table 3: The canFIS of CA. The second process is to generate minimal fault interaction set, as shown in Table 4 . The second column of Table 4 describes the interactions contained by canFIS in each step. The third column in Table 4 shows the interactions under testing. The fourth column shows the additional test cases for undertesting interactions. Columns 5 and 6 show the outputs of additional test cases and the set consisting of the elements deleted from canFIS, respectively. Table 4: The step of computing canFIS. We can conclude from Table 4 that the whole process takes 14 steps and each step generates an additional test case. The minimal fault interaction set is screened out at step (15) at last. Meanwhile we can get a conclusion that the number of steps the process takes depends on the order of interactions being tested. For example, if, in step (13) test interaction 22, the element to delete in canFIS is , then the minimal fault interaction set could be generated directly. The whole process only takes 13 steps and needs only 13 test cases. Therefore, an optimized interaction test order could reduce the generating of test cases. In this paper, we do not carry on the discussion and simply consider the order is random. The method presented by Ghandehari et al. [ 13 ] give the result that contains 9 interactions in the set; however, the minimal fault interaction set contains 2 interactions. This shows that the method comFIL is more precise than the method proposed by Ghandehari et al. 4. Empirical Study In algorithm comFIL, we need to record the number of each interaction existing in passed test cases and the number of each interaction existing in failed test cases. For a 1-way interaction, the value of is called fault ratio. The smaller the value is, the more probable the value is a safe value of the parameter. 4.1. Additional Test Case Generation In our experiment, the additional test case generation follows the following two steps: (I) For an input parameter , if its value is in interaction , then the value of in has the same value; otherwise, in is assigned by its value that has the smallest fault ratio. (II) Check whether belongs to ; if not, then is used as additional test case; otherwise, change a value of in till does not belong to . 4.2. Evaluation Criteria We use fault ratio as our evaluation criteria. In comFIL algorithm, we need to record the number of each interaction that exists in passed test cases and the number of each interaction that exists in failed test cases. For a 1-way interaction, the value of is called fault ratio. The smaller the value is, the more probable the value will be a safe value of the parameter. 4.3. Test Oracle Since the feature of these programs is not a concern in this paper, they are assumed to be correct. Then the standard and fault versions are compiled and run with test case as input; if the outputs of standard and fault versions are different, we believe the test case triggers the fault; that is, = fail; otherwise, = pass. 4.4. Experiment I (1) Experiment Objective . We use six C programs (comdline, count, nametbl, ntree, series, and tokens [ 12 ]) as test samples and input parameter model presented by Z. Zhang and J. Zhang [ 12 ]. Table 5 shows the basic information of these programs. Table 5: Test sample. The second column represents the number of lines without comments in each program, while column 3 refers to their input models. For example, means comdline has 9 parameters, in which 4 parameters have 3 values, 2 parameters have only 1 value, 1 parameter has 2 values, 1 parameter has 4 values, 2 parameters have 6 values, and 1 parameter has 15 values. Count (6; (2, 2, 3, 3, 3, 3)) can also be represented as . (2) Result of the Experiment . Table 6 indicates the detailed test results of the experiment; the data is mainly focused on test steps (additional test cases) and radix. In Table 6 , column 2 shows the number of test cases. Column 3 represents the different fault versions of each program. Column 4 shows the size of each canFIS. Column 5 refers to the number of test cases needed by each canFIS. Columns 6~11 represent the number of -way minimal fault interactions and the number of -way fault interactions to be selected, respectively; could be identified by the column title. For example, the first fault version of comdline is shown in Table 6 . It means that its canFIS’s size is 1663, and it needs 149 test cases. The numbers of 1~5-way minimal fault interactions are ; ; ; ; , respectively, and the minimal fault interaction larger than ways is . The numbers of interactions being tested for computing each are ; ; ; ; ; , respectively. Table 6: Test result of standard program. Table 7 is an experiment result compared with FIC algorithm presented by Z. Zhang and J. Zhang [ 12 ]. Table 7: Compare with FIC. Table 7 shows in most situations comFIL could generate a minor minimal fault interaction set compared to FIC. However, to locate the minimal fault interaction in software, comFIL need the involvement of more additional test cases than FIC algorithm. The main cause for that is that, for a test case that triggers the fault, it has child interaction; if the number of minimal fault interactions covers the array and test cases, we can not exclude the idea that most interactions of the child interaction set is fault interaction during the initial interaction fault detection. We may find that the result of ntree is not the minimal fault interaction set of . That is because many “assert” statements exist in its source codes. These statements make the program exit before data stream reaches the bug; that is, the program does not satisfy Assumption 2 . Therefore, the comFIL algorithm failed to identify the minimal fault interaction set of ntree. 4.5. Experiment II Generally, we consider a technique more effective if it could generate fewer test cases and be highly active. In this section, in order to verify the complexity of comFIL and show the additional test case count needs in different testing models, two simulated experiments will be examined. All the programs are developed with special design in order to better match the 3 assumptions of comFIL. (1) Simulated Experiment 1 . In this simulated experiment, we developed a simple program, Animal; it has an input model ; that is, the system has 7 parameters, and each of them has 5 values. We produce 5 fault versions by injecting fault in different position; each has 2~6 minimal fault interactions. The result of the experiment is shown in Figure 1 . The histogram in red shows the radix of canFIS while the data in green indicates the steps (additional test cases) to take to screen out the canFIS. Figure 1: Test result of Simulated Experiment 1. (a) Ratio of steps and radix. (b) Step and radix. (2) Simulated Experiment 2 . We developed 6 different programs which are much more complex than Animal in experiment 1; their input models are ; ; ; ; ; , respectively. Each program has 5 minimal fault interactions. The result of the experiment is shown in Figure 2 . Figure 2(a) shows the ratio of the number of additional test cases to the radix of canFIS. Figure 2(b) shows the result of experiment 2. Figure 2: Test result of Simulated Experiment 2. (a) Ratio of steps and radix. (b) Step and radix. The simulated experiment result presented by Figures 1 and 2 shows the number of additional test cases is decreasing while the program could better match the 3 assumptions of this paper. Furthermore, with the increment of minimal fault interactions’ count, the Ratio becomes smaller; this means the higher probability of getting safe value for input parameters. However, when the minimal fault interaction increases to a special value such as 9 in Figure 2(b) , the Ratio would rise; this is because when the number of input parameters becomes too large, too many additional test cases will be generated; this would affect the efficiency and lead the Ratio to rise. 4.6. Experiment Conclusion We can get the following conclusion about comFIL algorithm from both real program and simulated experiment results. (1) comFIL Could Obtain a Minimal Fault Interaction Set . From the result of experiment I—Table 7 , we can see that comFIL could generate a minor minimal fault interaction set compared to FIC. (2) comFIL Has Higher Capability of Getting Safe Value for Parameters . The result of experiment II shows, with the increment of minimal fault interactions’ count, the Ratio becomes smaller; this means the higher probability of getting safe value for input parameters. (3) The Efficiency of comFIL Would Be Affected While the Number of Input Parameters Is Too Large . When the number of input parameters is too large, the additional test cases size would be very huge; this is a limitation of comFIL, but we found that [ 14 ] presents an approach of leveraging the notion of inducing combination to locate the faults inside the source code; the combination of these two approaches would potentially benefit combinatorial testing. (4) The Assumptions Are Common but Sometimes They Do Not Establish in Real-World Programs . Although the assumptions of comFIL are common in combination testing study area, but sometimes they do not establish in real-world programs; this would limit the application of the algorithm and need to be further studied. 5. Conclusion and Future Work 5.1. Conclusion In this paper, we present a new combinatorial testing algorithm named comFIL, which could screen out the minimal fault interaction set of test cases. The main contributions of this paper are listed as follows: (1) Summarizing the basic idea of the previous fault interaction location techniques, including their advantages and disadvantages. (2) Proposing a novel fault interaction location method named comFIL (complete Fault Interaction Location) which has more powerful functionalities and performs more precisely compared with other fault location techniques. (3) Presenting the basic idea and theory model for comFIL and illustrating 2 key points when implementing comFIL. (4) Using 6 programs as sample and 2 additional simulated experiments to verify the precision and effectiveness of comFIL. If we can not obtain the safe value of each parameter before testing, the cost in generating an additional test case for an interaction is very high. However, if there are only a few bugs in a program, the number of minimal fault interactions is small and it is more probable for a parameter to be in its safe value. When we calculate the safe values of each parameter with proper methods, it is almost impossible that generated additional test cases do not satisfy Assumption 3 . Even for a test case generated for an interaction randomly, its possibility that it does not satisfy Assumption 3 is rather low. So almost every testing method in combinatorial testing could only work effectively when applied in program with relatively less faults. The theory and experiments indicate that comFIL is more accurate in fault localization compared with other algorithms in terms of combinatorial testing. However, comFIL also has its deficiency: (1) Assumption 3 is very strong and (2) the number of additional test cases to be generated is very large. 5.2. Future Work The future work will include three aspects: (1) for algorithm comFIL, lots of additional test cases should be generated; however, the order of the interactions being tested will influence the number of additional test cases generated. Thus optimizing the order of interactions being tested to reduce the number of additional test cases will be an interesting direction to explore further in the future. (2) The objective of algorithm comFIL is to generate the minimal fault interaction set, while how to use the minimal fault interaction set to further locate bugs will also be a significant topic to study further afterwards. (3) Exploring more effective combinatorial testing method according to different type of software (such as web service) is also worthwhile of further study. Competing Interests The authors declare that they have no competing interests. References J. D. Hagar, T. L. Wissink, D. R. Kuhn, and R. N. Kacker, “Introducing combinatorial testing in a large organization,” Computer , vol. 48, no. 4, pp. 64–72, 2015. View at Publisher · View at Google Scholar · View at Scopus W.-J. Zhou, D.-P. Zhang, and B.-W. Xu, “Locating error interactions based on partial covering array,” Chinese Journal of Computers , vol. 34, no. 6, pp. 1126–1136, 2011. View at Publisher · View at Google Scholar · View at Scopus C. Nie and H. Leung, “The minimal failure-causing schema of combinatorial testing,” ACM Transactions on Software Engineering and Methodology , vol. 20, no. 4, article 15, 2011. View at Publisher · View at Google Scholar · View at Scopus B. Garn and D. E. Simos, “Eris: a tool for combinatorial testing of the Linux system call interface,” in Proceedings of the 7th IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW '14) , pp. 58–67, Cleveland, Ohio, USA, April 2014. View at Publisher · View at Google Scholar · View at Scopus S. K. Khalsa and Y. Labiche, “An orchestrated survey of available algorithms and tools for combinatorial testing,” in Proceedings of the 25th IEEE International Symposium on Software Reliability Engineering (ISSRE '14) , pp. 323–334, IEEE, Naples, Italy, November 2014. View at Publisher · View at Google Scholar · View at Scopus A. Gupta and C. H. Scholz, “A model of normal fault interaction based on observations and theory,” Journal of Structural Geology , vol. 22, no. 7, pp. 865–879, 2000. View at Publisher · View at Google Scholar · View at Scopus D. C. P. Peacock, “Propagation, interaction and linkage in normal fault systems,” Earth-Science Reviews , vol. 58, no. 1-2, pp. 121–142, 2002. View at Publisher · View at Google Scholar · View at Scopus C. J. Colbourn and D. W. McClary, “Locating and detecting arrays for interaction faults,” Journal of Combinatorial Optimization , vol. 15, no. 1, pp. 17–48, 2008. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet · View at Scopus C. Martínez, L. Moura, D. Panario, et al., “Algorithms to locate errors using covering arrays,” in LATIN 2008: Theoretical Informatics , pp. 504–519, Springer, Berlin, Germany, 2008. View at Google Scholar J. Piton-Gonçalves and S. M. Aluísio, “An architecture for multidimensional computer adaptive test with educational purposes,” in Proceedings of the 18th Brazilian symposium on Multimedia and the web (WebMedia '12) , pp. 17–24, ACM, São Paulo, Brazil, 2012. View at Publisher · View at Google Scholar A. Zeller and R. Hildebrandt, “Simplifying and isolating failure-inducing input,” IEEE Transactions on Software Engineering , vol. 28, no. 2, pp. 183–200, 2002. View at Publisher · View at Google Scholar · View at Scopus Z. Zhang and J. Zhang, “Characterizing failure-causing parameter interactions by adaptive testing,” in Proceedings of the 20th International Symposium on Software Testing and Analysis (ISSTA '11) , pp. 331–341, ACM, Toronto, Canada, July 2011. View at Publisher · View at Google Scholar · View at Scopus L. Ghandehari, J. Chandrasekaran, Y. Lei, R. Kacker, and D. Kuhn, “Short paper: BEN: a combinatorial testing-based fault localization tool,” in Proceedings of the 4th International Workshop on Combinatorial Testing (in Junction with 8th IEEE International Conference on Software Testing, Verification and Validation) , April 2015. L. S. Ghandehari, Y. Lei, D. Kung, R. N. Kacker, and D. R. Kuhn, “Fault localization based on failure-inducing combinations,” in Proceedings of the 24th IEEE International Symposium on Software Reliability Engineering (ISSRE '13) , pp. 168–177, Pasadena, Calif, USA, November 2013. View at Publisher · View at Google Scholar · View at Scopus M. B. Cohen, M. B. Dwyer, and J. Shi, “Interaction testing of highly-configurable systems in the presence of constraints,” in Proceedings of the International Symposium on Software Testing and Analysis , pp. 129–139, ACM, 2007. (function (i, s, o, g, r, a, m) { i['GoogleAnalyticsObject'] = r; i[r] = i[r] || function () { (i[r].q = i[r].q || []).push(arguments) }, i[r].l = 1 * new Date(); a = s.createElement(o), m = s.getElementsByTagName(o)[0]; a.async = 1; a.src = g; m.parentNode.insertBefore(a, m) })(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga'); ga('create', 'UA-8578054-2', 'auto'); ga('send', 'pageview'); http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Advances in Software Engineering Hindawi Publishing Corporation

Locating Minimal Fault Interaction in Combinatorial Testing

Download PDF
11 pages

Loading next page...
 
/lp/hindawi-publishing-corporation/locating-minimal-fault-interaction-in-combinatorial-testing-QJz1gK5H9D

References

References for this paper are not available at this time. We will be adding them shortly, thank you for your patience.

Publisher
Hindawi Publishing Corporation
Copyright
Copyright © 2016 Wei Zheng et al.
ISSN
1687-8655
eISSN
1687-8663
Publisher site
See Article on Publisher Site

Abstract

Locating Minimal Fault Interaction in Combinatorial Testing div.banner_title_bkg div.trangle { border-color: #47132C transparent transparent transparent; opacity:0.8; /*new styles start*/ -ms-filter:"progid:DXImageTransform.Microsoft.Alpha(Opacity=80)" ;filter: alpha(opacity=80); /*new styles end*/ } div.banner_title_bkg_if div.trangle { border-color: transparent transparent #47132C transparent ; opacity:0.8; /*new styles start*/ -ms-filter:"progid:DXImageTransform.Microsoft.Alpha(Opacity=80)" ;filter: alpha(opacity=80); /*new styles end*/ } div.banner_title_bkg div.trangle { width: 317px; } #banner { background-image: url('http://images.hindawi.com/journals/ase/ase.banner.jpg'); background-position: 50% 0;} Hindawi Publishing Corporation Home Journals About Us Advances in Software Engineering About this Journal Submit a Manuscript Table of Contents Journal Menu About this Journal · Abstracting and Indexing · Aims and Scope · Article Processing Charges · Articles in Press · Author Guidelines · Bibliographic Information · Citations to this Journal · Contact Information · Editorial Board · Editorial Workflow · Free eTOC Alerts · Publication Ethics · Reviewers Acknowledgment · Submit a Manuscript · Subscription Information · Table of Contents Open Special Issues · Published Special Issues · Special Issue Guidelines Abstract Full-Text PDF Full-Text HTML Full-Text ePUB Full-Text XML Linked References How to Cite this Article Views 90 Citations 0 ePub 3 PDF 8 Advances in Software Engineering Volume 2016 (2016), Article ID 2409521, 10 pages http://dx.doi.org/10.1155/2016/2409521 Research Article Locating Minimal Fault Interaction in Combinatorial Testing Wei Zheng , Xiaoxue Wu , Desheng Hu , and Qihai Zhu College of Software & Micro-Electronics, Northwestern Polytechnical University, Xi’an 710072, China Received 14 October 2015; Revised 22 March 2016; Accepted 19 April 2016 Academic Editor: Henry Muccini Copyright © 2016 Wei Zheng et al. This is an open access article distributed under the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Abstract Combinatorial testing (CT) technique could significantly reduce testing cost and increase software system quality. By using the test suite generated by CT as input to conduct black-box testing towards a system, we are able to detect interactions that trigger the system’s faults. Given a test case, there may be only part of all its parameters relevant to the defects in system and the interaction constructed by those partial parameters is key factor of triggering fault. If we can locate those parameters accurately, this will facilitate the software diagnosing and testing process. This paper proposes a novel algorithm named complete Fault Interaction Location (comFIL) to locate those interactions that cause system’s failures and meanwhile obtains the minimal set of target interactions in test suite produced by CT. By applying this method, testers can analyze and locate the factors relevant to defects of system more precisely, thus making the process of software testing and debugging easier and more efficient. The results of our empirical study indicate that comFIL performs better compared with known fault location techniques in combinatorial testing because of its improved effectiveness and precision. 1. Introduction Combinatorial testing could significantly reduce test cost and increase quality of software system [ 1 ]. It has been proved to be effective especially in a software system where faults come from the interactions of its parameters [ 2 ]. Combinatorial testing could detect the parameter interactions that trigger the system faults rather than localize it. If a test case triggers the fault of a system, it reflects that there exists one or more defects in the program [ 3 – 5 ]. However, not all parameters in the test case are relevant to defects. If we are able to locate a parameter in the test case that is relevant to the fault, we can apply this useful information to facilitate the debugging process. In combinatorial testing, the study on fault interaction technique could be categorized into adaptive method and nonadaptive method according to the dependence between additional test cases and running results [ 6 , 7 ]. For nonadaptive methods, the generation of additional test cases does not rely on the running result of original test cases. Colbourn and McClary [ 8 ] present a nonadaptive method named Locating and Detecting Arrays (LDA). Based on basic known information such as parameters’ number, values, and faults’ number, the method applies -way Locating and Detecting Array to locate faults in software system. Martínez et al. [ 9 ] present a self-adaptive algorithm based on Errors Locating Arrays (ELA) and analyze the algorithm complexity. However, this method could only be used under the condition that the value’s number of each parameter in software is not larger than 2. On the basis of LDA and ELA, Hagar et al. [ 1 ] propose the method of Partial Covering Array (PCA) which could be used in the software with known safe value, and it presents a new combinatorial structure to generate ELA. Another category is known as adaptive method [ 10 ], whose generation of additional test cases depends on the information given by the execution of original test cases. Zeller and Hildebrandt [ 11 ] present a typical adaptive method named Delta Debugging. The main idea of this method is to identify the interaction that is relevant to the faults by modifying the input parameters. For a test case that triggers the fault, modify some of its input parameters; if the modified test case still triggers the fault, then the modified parameters are irrelevant to fault; otherwise, the modified parameters are related to fault. Based on Delta Debugging, Z. Zhang and J. Zhang [ 12 ] present a method named FIC. Similar to Delta Debugging, FIC modify one parameter in a test case with parameters once. Then repeat this process times and the minimal fault interaction is calculated afterwards. A restraint of Delta Debugging based methods is that they could only be applicable to the test case containing one minimal fault interaction, but, in real-world programs, more than one minimal fault interaction is very common. Ghandehari et al. [ 13 ] present a fault localization tool based on a failure-inducing combinations algorithm of [ 14 ]; it leverages the notion of inducing combination to locate the faults inside the source code. To locate each interaction related to faults, in this paper we present a new complete Fault Interaction Location (comFIL) method. The method includes 2 steps to locate minimal fault interactions. First, after the execution of original test cases, the test cases will be divided into 2 sets: FTS and PTS; the former contains test cases that trigger faults, while the latter contains test cases that do not trigger faults. Then we identify the set of interactions covered by FTS but not covered by PTS and name this set as candidate faulty interaction set (canFIS). Second, we generate additional test cases to select interactions in canFIS and then the minimal fault interaction set is obtained finally. 2. Preliminaries 2.1. Basic Definitions Assume there is a system under test (SUT) with input parameters as . For any , the range of parameter is denoted as . And pair is used to indicate that the value of parameter is , also denoted as ( ). For example, a model named BCSD is shown in Table 1 . The model has 4 input parameters: client browser, client operating system (client OS), server operating system (server OS), and server database. Table 1: The input model of a simple system. In this model, the range of parameter is . The input parameter model of SUT is denoted as , which means SUT has input parameters and, for each parameter , the value numbers are , respectively. Definition 1 ( -way interaction). For a set of elements , if for any , there are , then we call a -way interaction. Definition 2 (superinteraction and subinteraction). For 2 interactions, and , if , then we call the subinteraction of and the superinteraction of . An interaction is both the subinteraction and superinteraction of itself. For example, in model BSCD, interaction is the subinteraction of interaction . Definition 3 (subinteraction set). For a -way interaction , its power set is ; is denoted as the set of all subinteractions of . Definition 4 (fault interaction). For an interaction , if , then is a fault interaction. Definition 5 (minimal fault interaction). For an interaction , if for any interaction , is not fault interaction, then is minimal fault interaction. Definition 6 (additional test case function). For an interaction , additional test case function generates a test case that satisfies the following: if , then is a fault interaction; otherwise, is not a fault interaction, denoted as . Definition 7 (candidate fault interaction set). Denote as a candidate fault interaction set of . 2.2. Basic Assumptions Assumption 1. The parameters in the system are independent of each other. In many systems, there are constraint conditions between parameters. In [ 15 ] Cohen et al. study the method to generate test cases under constraint conditions. However, the constraint conditions among parameters are not the focus of fault location method, so we do not consider the dependence between parameters in this paper. Assumption 2. If an interaction causes the fault of the system, the test cases containing this interaction will trigger the fault necessarily. Assumption 3. The test cases generated by additional test case function do not contain the minimal fault interaction. For an interaction , generate a test case ; if trigger the fault, then this fault is caused by interactions in the set of subinteractions of . If contains the minimal fault interaction, the complexity of fault location will be increased. So we avoid considering it in this paper. This assumption will be explained later in the implementation of . The algorithm comFIL presented by this paper is based on the assumptions above. The algorithm is effective only under the condition that all these 3 assumptions are tenable. 2.3. Basic Inferences Inference 1. The test case that contains fault interaction will trigger the fault of system necessarily and the test case that does not trigger the fault does not contain the fault interaction. Inference 2. If an interaction is not a fault interaction, any of its subinteractions is not fault interaction. 2.4. Basic Theorems Theorem 1. For 2 interactions and , if , then . Proof. From Definition 3 (subinteraction set), we can know that and for any , , so ; namely, . Theorem 2. Divide into 2 sets: FTS and PTS. In FTS, all test cases triggered the fault of the system, ; if is the minimal fault interaction of , then . Proof. From Inference 1 we can know that, for any minimal fault interaction in , there is a failed test case that satisfies . So and ; that is, . Theorem 3. Use MFIS which denotes all minimal fault interactions set of ; for a minimal fault interaction , if and , then and . Proof. This could be directly concluded from Theorem 2 . Theorem 4. For a -way interaction , the additional test case is . In subinteraction set of , the number of interactions that do not belong to is . Proof. From Definition 4 we know that and , in which is the subinteraction of . From Theorem 1 we know that . So . 3. comFIL Algorithm 3.1. Description of comFIL Theorem 3 and Definition 5 provide a screening method for obtaining the set of all minimal fault interactions from the set of candidate fault interactions; we name the set as complete Fault Interaction Location and denote it as comFIL for short. When an interaction proves to be the fault interaction, we delete all the parent interactions of from canFIS except , while we delete all the child interactions of from canFIS except when it proves not to be the fault interaction. Under the three assumptions in Section 2 , algorithm comFIL is based on Theorems 2 and 3 and Definition 6 as theoretical core. According to Theorem 2 , we calculate the set of candidate fault interactions from the combination test cases. And, based on Theorem 3 and Definition 6 , we screen out the set of minimal fault interactions from the set of candidate fault interactions. The basic framework of comFIL algorithm is shown in Algorithm 1 . It has two input parameters and and an output parameter canFIS, which is the minimal fault interaction set. Generally the algorithm process could be divided into 2 phases. Algorithm 1: The comFIL algorithm. Generate the canFIS for fault location. Firstly, the algorithm counts the number of times that the fault interaction exists in passed or failed test cases separately (steps (2) to (12)); then the canFIS is screened out from test case set (steps (13) to (15)). Generate the minimal fault interaction set (steps (16) to (20)). Steps (17) and (18) describe the following: if a schema is a fault interaction, then we delete all its superinteraction except for in canFIS. Steps (19) and (20) describe the following: if an interaction is not a fault interaction, then we delete all elements in its subinteraction in canFIS. 3.2. Implementation of Key Functions of Algorithm comFIL (1) Implementation of subSet . The input of function subSet is an interaction , and its output is the subinteraction set of . For -way interaction , its -way subinteraction is a binary string , in which ; that is, a -way subinteraction of is , in which are not equal to each other. For example, a subinteraction of interaction is , whose binary string is . A binary string is corresponding to a certain decimal number, so, for -way interaction , its subinteraction set is the corresponding binary stings of integer set . (2) Implementation of addTF . The input of function addTF is an interaction , and its output is an additional test case . For the input , the number of its ways is not ; that is, is not a test case. From Assumption 3 we can know that ; moreover, is the only minimal fault interaction among all subinteractions of . From Theorem 4 we know that when generating addition test case , we will first ensure that there is no minimal fault interaction in subinteractions. However, if is very large and is relatively small, the numbers of interactions to be examined and additional test cases to be generated are very large. For convenience, we assume each parameter has a value, which does not associate with any fault (i.e., this value does not belong to any minimal fault interaction). This value is denoted as safe value. In comFIL, the number of times that each interaction appears in passed test cases and failed test cases needs to be recorded. For a one-way interaction, that is, a value of a parameter, the value is named the fault ratio of the value. We simply consider that the smaller the fault ratio of a parameter’s value is, the more likely this value would be the safe value of the parameter. When it comes to testing an object as we could not get the safe value of each parameter before testing, according to Theorem 4 , normally there will be an enormous number of interactions to be tested for the additional test cases generated by the interaction, resulting in the fact that the cost is very high. However, when there only exists a small amount of software defects (e.g., the software in the Delta testing phase), the number of the minimal fault interactions in the combination test cases is very small and thus there is high possibility for the existence of safe value in the parameters. The process of generating additional test cases is as follows. First, if a value of a parameter appears in interaction , the value of in additional test case is itself; otherwise it will be assigned by the value that has the smallest safe value. If many values of have the same smallest safe value, then will be assigned randomly among these values. Second, to check , if it does not belong to test suite , then is used as an additional test case; otherwise, will be regenerated. The regeneration process is to modify a parameter’s value in by assigning this parameter another value whose safe value is the smallest or second smallest, thus making sure is a subinteraction of . Then we repeat this process till does not belong to . 3.3. An Application of comFIL As shown in Algorithm 2 , foo is a method proposed in [ 13 ]. The correct prototype of foo in line (11) should be “ ”; because of the loss of “+” it becomes the wrong statement “ ,” and it is a bug of program foo. The model of foo’s input parameters is SUT (4; (2, 2, 3, 4)). The ranges of parameters are ; ; ; , respectively. If an input contains or , data stream could reach the bug of foo; meanwhile, the state of foo is different from its correct prototype. Therefore, the minimal fault interaction set is . Algorithm 2: A fault program. The program foo is tested on 2-way coverage in this paper. Test case set and test result are shown in Table 2 . The algorithm comFIL uses Table 2 as input. The first process is to screen out canFIS from Table 2 . Table 2: Test result of 2-way coverage. Table 3 shows the results of first process, in which the first column refers to the number of interactions while the first row represents the parameter of each interaction and the remaining rows represent the respective value of the parameters. Each row in Table 3 indicates an interaction from row 2 on. For example, interaction 1 is shown in row 2 (I # 1). Table 3: The canFIS of CA. The second process is to generate minimal fault interaction set, as shown in Table 4 . The second column of Table 4 describes the interactions contained by canFIS in each step. The third column in Table 4 shows the interactions under testing. The fourth column shows the additional test cases for undertesting interactions. Columns 5 and 6 show the outputs of additional test cases and the set consisting of the elements deleted from canFIS, respectively. Table 4: The step of computing canFIS. We can conclude from Table 4 that the whole process takes 14 steps and each step generates an additional test case. The minimal fault interaction set is screened out at step (15) at last. Meanwhile we can get a conclusion that the number of steps the process takes depends on the order of interactions being tested. For example, if, in step (13) test interaction 22, the element to delete in canFIS is , then the minimal fault interaction set could be generated directly. The whole process only takes 13 steps and needs only 13 test cases. Therefore, an optimized interaction test order could reduce the generating of test cases. In this paper, we do not carry on the discussion and simply consider the order is random. The method presented by Ghandehari et al. [ 13 ] give the result that contains 9 interactions in the set; however, the minimal fault interaction set contains 2 interactions. This shows that the method comFIL is more precise than the method proposed by Ghandehari et al. 4. Empirical Study In algorithm comFIL, we need to record the number of each interaction existing in passed test cases and the number of each interaction existing in failed test cases. For a 1-way interaction, the value of is called fault ratio. The smaller the value is, the more probable the value is a safe value of the parameter. 4.1. Additional Test Case Generation In our experiment, the additional test case generation follows the following two steps: (I) For an input parameter , if its value is in interaction , then the value of in has the same value; otherwise, in is assigned by its value that has the smallest fault ratio. (II) Check whether belongs to ; if not, then is used as additional test case; otherwise, change a value of in till does not belong to . 4.2. Evaluation Criteria We use fault ratio as our evaluation criteria. In comFIL algorithm, we need to record the number of each interaction that exists in passed test cases and the number of each interaction that exists in failed test cases. For a 1-way interaction, the value of is called fault ratio. The smaller the value is, the more probable the value will be a safe value of the parameter. 4.3. Test Oracle Since the feature of these programs is not a concern in this paper, they are assumed to be correct. Then the standard and fault versions are compiled and run with test case as input; if the outputs of standard and fault versions are different, we believe the test case triggers the fault; that is, = fail; otherwise, = pass. 4.4. Experiment I (1) Experiment Objective . We use six C programs (comdline, count, nametbl, ntree, series, and tokens [ 12 ]) as test samples and input parameter model presented by Z. Zhang and J. Zhang [ 12 ]. Table 5 shows the basic information of these programs. Table 5: Test sample. The second column represents the number of lines without comments in each program, while column 3 refers to their input models. For example, means comdline has 9 parameters, in which 4 parameters have 3 values, 2 parameters have only 1 value, 1 parameter has 2 values, 1 parameter has 4 values, 2 parameters have 6 values, and 1 parameter has 15 values. Count (6; (2, 2, 3, 3, 3, 3)) can also be represented as . (2) Result of the Experiment . Table 6 indicates the detailed test results of the experiment; the data is mainly focused on test steps (additional test cases) and radix. In Table 6 , column 2 shows the number of test cases. Column 3 represents the different fault versions of each program. Column 4 shows the size of each canFIS. Column 5 refers to the number of test cases needed by each canFIS. Columns 6~11 represent the number of -way minimal fault interactions and the number of -way fault interactions to be selected, respectively; could be identified by the column title. For example, the first fault version of comdline is shown in Table 6 . It means that its canFIS’s size is 1663, and it needs 149 test cases. The numbers of 1~5-way minimal fault interactions are ; ; ; ; , respectively, and the minimal fault interaction larger than ways is . The numbers of interactions being tested for computing each are ; ; ; ; ; , respectively. Table 6: Test result of standard program. Table 7 is an experiment result compared with FIC algorithm presented by Z. Zhang and J. Zhang [ 12 ]. Table 7: Compare with FIC. Table 7 shows in most situations comFIL could generate a minor minimal fault interaction set compared to FIC. However, to locate the minimal fault interaction in software, comFIL need the involvement of more additional test cases than FIC algorithm. The main cause for that is that, for a test case that triggers the fault, it has child interaction; if the number of minimal fault interactions covers the array and test cases, we can not exclude the idea that most interactions of the child interaction set is fault interaction during the initial interaction fault detection. We may find that the result of ntree is not the minimal fault interaction set of . That is because many “assert” statements exist in its source codes. These statements make the program exit before data stream reaches the bug; that is, the program does not satisfy Assumption 2 . Therefore, the comFIL algorithm failed to identify the minimal fault interaction set of ntree. 4.5. Experiment II Generally, we consider a technique more effective if it could generate fewer test cases and be highly active. In this section, in order to verify the complexity of comFIL and show the additional test case count needs in different testing models, two simulated experiments will be examined. All the programs are developed with special design in order to better match the 3 assumptions of comFIL. (1) Simulated Experiment 1 . In this simulated experiment, we developed a simple program, Animal; it has an input model ; that is, the system has 7 parameters, and each of them has 5 values. We produce 5 fault versions by injecting fault in different position; each has 2~6 minimal fault interactions. The result of the experiment is shown in Figure 1 . The histogram in red shows the radix of canFIS while the data in green indicates the steps (additional test cases) to take to screen out the canFIS. Figure 1: Test result of Simulated Experiment 1. (a) Ratio of steps and radix. (b) Step and radix. (2) Simulated Experiment 2 . We developed 6 different programs which are much more complex than Animal in experiment 1; their input models are ; ; ; ; ; , respectively. Each program has 5 minimal fault interactions. The result of the experiment is shown in Figure 2 . Figure 2(a) shows the ratio of the number of additional test cases to the radix of canFIS. Figure 2(b) shows the result of experiment 2. Figure 2: Test result of Simulated Experiment 2. (a) Ratio of steps and radix. (b) Step and radix. The simulated experiment result presented by Figures 1 and 2 shows the number of additional test cases is decreasing while the program could better match the 3 assumptions of this paper. Furthermore, with the increment of minimal fault interactions’ count, the Ratio becomes smaller; this means the higher probability of getting safe value for input parameters. However, when the minimal fault interaction increases to a special value such as 9 in Figure 2(b) , the Ratio would rise; this is because when the number of input parameters becomes too large, too many additional test cases will be generated; this would affect the efficiency and lead the Ratio to rise. 4.6. Experiment Conclusion We can get the following conclusion about comFIL algorithm from both real program and simulated experiment results. (1) comFIL Could Obtain a Minimal Fault Interaction Set . From the result of experiment I—Table 7 , we can see that comFIL could generate a minor minimal fault interaction set compared to FIC. (2) comFIL Has Higher Capability of Getting Safe Value for Parameters . The result of experiment II shows, with the increment of minimal fault interactions’ count, the Ratio becomes smaller; this means the higher probability of getting safe value for input parameters. (3) The Efficiency of comFIL Would Be Affected While the Number of Input Parameters Is Too Large . When the number of input parameters is too large, the additional test cases size would be very huge; this is a limitation of comFIL, but we found that [ 14 ] presents an approach of leveraging the notion of inducing combination to locate the faults inside the source code; the combination of these two approaches would potentially benefit combinatorial testing. (4) The Assumptions Are Common but Sometimes They Do Not Establish in Real-World Programs . Although the assumptions of comFIL are common in combination testing study area, but sometimes they do not establish in real-world programs; this would limit the application of the algorithm and need to be further studied. 5. Conclusion and Future Work 5.1. Conclusion In this paper, we present a new combinatorial testing algorithm named comFIL, which could screen out the minimal fault interaction set of test cases. The main contributions of this paper are listed as follows: (1) Summarizing the basic idea of the previous fault interaction location techniques, including their advantages and disadvantages. (2) Proposing a novel fault interaction location method named comFIL (complete Fault Interaction Location) which has more powerful functionalities and performs more precisely compared with other fault location techniques. (3) Presenting the basic idea and theory model for comFIL and illustrating 2 key points when implementing comFIL. (4) Using 6 programs as sample and 2 additional simulated experiments to verify the precision and effectiveness of comFIL. If we can not obtain the safe value of each parameter before testing, the cost in generating an additional test case for an interaction is very high. However, if there are only a few bugs in a program, the number of minimal fault interactions is small and it is more probable for a parameter to be in its safe value. When we calculate the safe values of each parameter with proper methods, it is almost impossible that generated additional test cases do not satisfy Assumption 3 . Even for a test case generated for an interaction randomly, its possibility that it does not satisfy Assumption 3 is rather low. So almost every testing method in combinatorial testing could only work effectively when applied in program with relatively less faults. The theory and experiments indicate that comFIL is more accurate in fault localization compared with other algorithms in terms of combinatorial testing. However, comFIL also has its deficiency: (1) Assumption 3 is very strong and (2) the number of additional test cases to be generated is very large. 5.2. Future Work The future work will include three aspects: (1) for algorithm comFIL, lots of additional test cases should be generated; however, the order of the interactions being tested will influence the number of additional test cases generated. Thus optimizing the order of interactions being tested to reduce the number of additional test cases will be an interesting direction to explore further in the future. (2) The objective of algorithm comFIL is to generate the minimal fault interaction set, while how to use the minimal fault interaction set to further locate bugs will also be a significant topic to study further afterwards. (3) Exploring more effective combinatorial testing method according to different type of software (such as web service) is also worthwhile of further study. Competing Interests The authors declare that they have no competing interests. References J. D. Hagar, T. L. Wissink, D. R. Kuhn, and R. N. Kacker, “Introducing combinatorial testing in a large organization,” Computer , vol. 48, no. 4, pp. 64–72, 2015. View at Publisher · View at Google Scholar · View at Scopus W.-J. Zhou, D.-P. Zhang, and B.-W. Xu, “Locating error interactions based on partial covering array,” Chinese Journal of Computers , vol. 34, no. 6, pp. 1126–1136, 2011. View at Publisher · View at Google Scholar · View at Scopus C. Nie and H. Leung, “The minimal failure-causing schema of combinatorial testing,” ACM Transactions on Software Engineering and Methodology , vol. 20, no. 4, article 15, 2011. View at Publisher · View at Google Scholar · View at Scopus B. Garn and D. E. Simos, “Eris: a tool for combinatorial testing of the Linux system call interface,” in Proceedings of the 7th IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW '14) , pp. 58–67, Cleveland, Ohio, USA, April 2014. View at Publisher · View at Google Scholar · View at Scopus S. K. Khalsa and Y. Labiche, “An orchestrated survey of available algorithms and tools for combinatorial testing,” in Proceedings of the 25th IEEE International Symposium on Software Reliability Engineering (ISSRE '14) , pp. 323–334, IEEE, Naples, Italy, November 2014. View at Publisher · View at Google Scholar · View at Scopus A. Gupta and C. H. Scholz, “A model of normal fault interaction based on observations and theory,” Journal of Structural Geology , vol. 22, no. 7, pp. 865–879, 2000. View at Publisher · View at Google Scholar · View at Scopus D. C. P. Peacock, “Propagation, interaction and linkage in normal fault systems,” Earth-Science Reviews , vol. 58, no. 1-2, pp. 121–142, 2002. View at Publisher · View at Google Scholar · View at Scopus C. J. Colbourn and D. W. McClary, “Locating and detecting arrays for interaction faults,” Journal of Combinatorial Optimization , vol. 15, no. 1, pp. 17–48, 2008. View at Publisher · View at Google Scholar · View at Zentralblatt MATH · View at MathSciNet · View at Scopus C. Martínez, L. Moura, D. Panario, et al., “Algorithms to locate errors using covering arrays,” in LATIN 2008: Theoretical Informatics , pp. 504–519, Springer, Berlin, Germany, 2008. View at Google Scholar J. Piton-Gonçalves and S. M. Aluísio, “An architecture for multidimensional computer adaptive test with educational purposes,” in Proceedings of the 18th Brazilian symposium on Multimedia and the web (WebMedia '12) , pp. 17–24, ACM, São Paulo, Brazil, 2012. View at Publisher · View at Google Scholar A. Zeller and R. Hildebrandt, “Simplifying and isolating failure-inducing input,” IEEE Transactions on Software Engineering , vol. 28, no. 2, pp. 183–200, 2002. View at Publisher · View at Google Scholar · View at Scopus Z. Zhang and J. Zhang, “Characterizing failure-causing parameter interactions by adaptive testing,” in Proceedings of the 20th International Symposium on Software Testing and Analysis (ISSTA '11) , pp. 331–341, ACM, Toronto, Canada, July 2011. View at Publisher · View at Google Scholar · View at Scopus L. Ghandehari, J. Chandrasekaran, Y. Lei, R. Kacker, and D. Kuhn, “Short paper: BEN: a combinatorial testing-based fault localization tool,” in Proceedings of the 4th International Workshop on Combinatorial Testing (in Junction with 8th IEEE International Conference on Software Testing, Verification and Validation) , April 2015. L. S. Ghandehari, Y. Lei, D. Kung, R. N. Kacker, and D. R. Kuhn, “Fault localization based on failure-inducing combinations,” in Proceedings of the 24th IEEE International Symposium on Software Reliability Engineering (ISSRE '13) , pp. 168–177, Pasadena, Calif, USA, November 2013. View at Publisher · View at Google Scholar · View at Scopus M. B. Cohen, M. B. Dwyer, and J. Shi, “Interaction testing of highly-configurable systems in the presence of constraints,” in Proceedings of the International Symposium on Software Testing and Analysis , pp. 129–139, ACM, 2007. (function (i, s, o, g, r, a, m) { i['GoogleAnalyticsObject'] = r; i[r] = i[r] || function () { (i[r].q = i[r].q || []).push(arguments) }, i[r].l = 1 * new Date(); a = s.createElement(o), m = s.getElementsByTagName(o)[0]; a.async = 1; a.src = g; m.parentNode.insertBefore(a, m) })(window, document, 'script', '//www.google-analytics.com/analytics.js', 'ga'); ga('create', 'UA-8578054-2', 'auto'); ga('send', 'pageview');

Journal

Advances in Software Engineering – Hindawi Publishing Corporation

Published: May 29, 2016

There are no references for this article.