Testing embedded system through optimal mining technique (OMT) based on multi-input domain

Testing embedded systems must be done carefully particularly in the significant regions of the embedded systems. Inputs from an embedded system can happen in multiple order and many relationships can exist among the input sequences. Consideration of the sequences and the relationships among the sequences is one of the most important considerations that must be tested to find the expected behavior of the embedded systems. On the other hand combinatorial approaches help determining fewer test cases that are quite enough to test the embedded systems exhaustively. In this paper, an Optimal Mining Technique that considers multi-input domain which is based on built-in combinatorial approaches has been presented. The method exploits multi-input sequences and the relationships that exist among multi-input vectors. The technique has been used for testing an embedded system that monitors and controls the temperature within the Nuclear reactor s. helps address this ask by placing Combinatorial Test Design (CTD) at the heart of the solution. This document presents two case studies of CTD implementation in client engagements and focuses on the approach, process and challenges addressed to scale up the implementation and make CTD a mainstream activity. The IBM Focus tool was used in both cases to implement Combinatorial Test Design for optimization of tests and for reducing test effort while increasing test coverage.


INTRODUCTION 1.Background
Testing is playing a significant role in the development of any system which is a systematized process to verify the reliability, behaviour and performance of a system against considered stipulations. It enables a device or a system to be as defect-free as possible which act as a one of the detective measures, and verification is one of the corrective measures of quality.
Black-box testing inspects the functionality of an application without seeing into its internal structures or workings. It mainly concentrates on the functional requirements of the embedded system without considering the internal working of the system. The main aim of this testing is to select the acceptable test cases and detect as many faults based on requirements specification at least cost and time.
Testing embedded systems involves testing software, hardware and both. Testing of hardware and software however can be conducted independently and then the testing has to be undertaken after the software is migrated into the hardware.
Embedded systems are a mixture of various computing devices, such as microcontrollers, application-specific integrated circuits, and digital signal processors. Some widely used systems in real world applications such as routers, power plant system, medical instrument system, home appliances, air traffic control station, and firewalls, telecommunication exchanges, robotics, industrial automation and smart cards etc. are example of embedded system. Falseness in hardware systems may be designated in terms of defect, error and faults. Combinatorial testing is commonly utilized black-box practice that could dramatically diminish the number of test cases, as it is a highly competent technique to perceive software faults. This method originates test cases from input domain of the system under test. But, when the input domain is noteworthy and the output domain is much tiny, it is desirable to go for testing the output domain either exhaustively [1] or as much as possible.
For a few safety critical embedded systems, building test cases drawn from multi-input domain is a necessity as multiple inputs can occur at the same time. However the an embedded system must also be tested form other perspectives that include input, output, input-output and Multi-output domain Generation of test cases based on Multi-input perspective will be more suitable than other perspective's as it guarantees that all or as many possible input combinations are comprehensively tested.
Exhaustive testing [2] considering non multi-input domains is out of question when many input variables exist and they act in several combinations. Pseudo-Exhaustive testing aims at considering only those combinations that will most likely result in failure conditions.
Optimal Mining Technique (OMT) derives test cases by choosing certain combinations of either the inputs or outputs based on the possibility of occurrence of multi-output or multi-input domain of an embedded system such as TMCNRS which monitors and controls temperatures within nuclear reactor systems.
In the case of TMCNRS, the occurrences temperatures within nuclear records can happen simultaneously are independently. In an embedded system processing tasks are designed for handling multiinputs which occur simultaneously. These tasks must be tested thoroughly to guarantee the proper working of the embedded system. Test cases should be generated to verify the functionality of occurrence of proper outputs based on the Occurrence of Multi-input.

Problem
In the case of embedded systems, Inputs occur as a set in addition to the occurrence of independent inputs. The behaviour of an embedded system when multi-inputs occurs must be tested to find the whether the system has been properly developed to process multi-inputs that occur simultaneously.

Proposed solution
An improved optimal Mining technique is presented in this paper, the range of values that must be used for generating the test data have also been pre-identified and mapped with output variables. The multi input relationships and also the input output relationships which can be used as a database for mining relationship pattern for further modelling. The pattern of occurrence of the input variable can be determined by using a mining algorithm or through manual inspection.

RELATED WORKS
Lakshmi Prasad, et al., [3] had presented a comprehensive survey on combinatorial testing. In [4] a method is proposed that deals with generation of test cases based on the input domain considering with special consideration to testing standalone embedded system. The algorithm can successfully generate pairs for those input parameters and eliminate non related input pairs thereby reducing the size of the test suite to a minimum. Lakshmi Prasad, et al., [5] had proposed several combinatorial methods for testing an embedded system.
Gray. D. M. Cohen, et al., [6] introduced the combinatorial design approach for generating the test cases automatically. They have described an application which is developed using the method presented by them. They have shown that the time required for the development of test plan has been reduced considerably and they have also shown that the entire code has been covered using the test plan that has been used to generate test cases.
Cohen D. M., et al., [7] have presented a system called AETG. The approach presented by them considers all combinations of input parameters that include pair-wise, tripe-wise and n-wise. The approaches presented by them will breed all the valid test pairs ignoring the invalid test pairs. The numbers of test cases generally are of the logarithmic order of the number of input variables used. The AETG has been used for undertaking different types of testing that include unit, functional, acceptance, system, integration, regression and inter-operability testing.
Cohen D. M., [8] have presented a method and a system for enumerating a minimal number of test cases for systems with interacting elements that have relationships between the elements and the number of characteristics evaluated for each element. In the method, the user enters values for each of the elements and then defines relationships between the elements. Our method then enumerates a table of test cases for each relationship between elements using deterministic procedures, when applicable, and random procedures The significant expansion of autonomous control and information processing capabilities in the coming generation of mission software systems results in a qualitatively larger space of behaviours that needs to be "covered" during testing, not only at the system level but also at subsystem and unit levels Tung, [9]. A major challenge in this area is to automatically generate a relatively small set of test cases that, collectively, guarantees a selected degree of coverage of the behaviour space. They described an algorithm for a parametric test case generation tool that applies a combinatorial design approach to the selection of candidate test cases. Evaluation of this algorithm on test parameters from the Deep Space One mission reveals a valuable reduction in the number of test cases, when compared to an earlier home-brewed generator.
Lei Y., et al., [10] have used a criterion which is test specification based. The criterion considers each pair of input variables, and every value pair of the input pair considered and every value selected covered through a test case. They have evolved this strategy for carrying pair-wise testing.
Covering arrays have been augmented further by Cohen M. B., et al., [11] through inclusion of the concept called Annealing. This has led to special array containing several sub-arrays which all together contain all the t-tuples, each tuple appearing at least once. The strength of the array is measured through t number of tuples contained in the array considering all the sub-array contained in it. They have analyzed all the arrays that have strength of 3 tuples using recursive combinatorial construction and using search techniques. The technique used by them leveraged optimality and efficiency of the size through use of combinatorial construction and heuristic search.
Further addition to the recursive combinatorial generation added with heuristic search has been made through detection of interaction of multiple components that lead to different kinds of failures. R. Kuhn, et al., [12] have applied this approach to the real world applications and obtained the analytical results.
The way a system is tested depended on the type of the system. Choice of an appropriate testing method is crucial for making testing effective and rational. The application related to flow of water inside a carbon Nano tube that is single walled and in which several temperature gradients exist has been tested by Shiomi J., et al., [13] by using combinatorial method.
Ochoa, et al., [14] had introduced the box-fusion which is an approach to improve pair wise testing. Box-Fusion approach was guessed and a case study was carried out by using two software implementations: the Simple LTL Generator that builds Linear Temporal Logic (LTL) formulae with atomic propositions and the prospect algorithm that can produce LTL formulae from more than 31,000 possible input combinations. They have presented evaluation of Box-Fusion approach which considers, pair wise testing approach, annotated control flow graphs approach and regression testing approach.
M. Lakshmi Prasad, et al., [15]- [19] had built test cases by particle swarm optimization (PSO) for multi output domain embedded systems using combinatorial techniques. They also used neural network based strategy for automated construction of test cases for testing an embedded system using combinatorial techniques. They also developed generating test cases for testing web sites through neural networks and input pairs. They also generated test cases using combinatorial methods based multi-output domain of an embedded system through the process of optimal selection. Abdul Rahman, et al., [20] had presented a survey on input-output relationship relation to test data generation strategies. They reviewed the existing combinatorial test data generation strategies supporting the IOR features specifically taking the nature inspired algorithm as the main basis. Benchmarking results illustrate the comparative performance of existing nature inspired algorithm based strategies supporting IOR.
Combinatorial methods can also be used for testing software that predominantly uses logical expressions based on Boolean or binary inputs. In the software used related to most of the safety critical applications, Boolean expressions are used extensively. S. Vilkomir, [21] has steadied effectiveness of combinatorial testing when binary inputs are used.
Deepa Gupta, et al., [22] had proposed a sequence generation of test cases using pair wise approach. They presented an approach which uses the series origination approach for pair wise test case origination. This approach makes certain to disseminate the required intent of trial run cases which cover all available relations between all instructions pairs at least once. Trial run selection specification is this approach is based on combinatorial testing.
Jose Torres-Jimenez, et al., [23] Covering arrays are combinatorial structures which have applications in fields like software testing and hardware Trojan detection. In this paper we proposed a two-stage simulated annealing algorithm to construct covering arrays. The proposed algorithm is instanced in this paper through the construction of ternary covering arrays of strength three. We were able to get 579 new upper bounds. In order to show the generality of our proposal, we defined a new benchmark composed of 25 instances of MCAs taken from the literature, all instances were improved.  [24] Clients today want more for less and the IBM test mantra of Test Less Test Right helps address this ask by placing Combinatorial Test Design (CTD) at the heart of the solution. This document presents two case studies of CTD implementation in client engagements and focuses on the approach, process and challenges addressed to scale up the implementation and make CTD a mainstream activity. The IBM Focus tool was used in both cases to implement Combinatorial Test Design for optimization of tests and for reducing test effort while increasing test coverage. P. S., M. B., et al., [25] Combinatorial Testing is a test design methodology that aims to detect the interaction failures existing in the software under test. The combinatorial input space model comprises of the parameters and the values it can take. Building this input space model is a domain knowledge and experience intensive task. The objective of the paper is to assist test designer in building this test model. A rule based semi-automatic approach is proposed to derive the input space model elements from Use case specifications and UML use case diagrams. A natural language processing based parser and an XMI based parser are implemented. The rules formulated are applied on synthetic case studies and the output model is evaluated using precision and recall metrics. The results are promising and this approach will be of good use to the test designer.
Y. Yao, et al., [26] as an effective software testing technique, combinatorial testing has been gradually applied in various types of test practice. In this case, it is necessary to provide useful combinatorial testing tools to support the application of combinatorial testing technique on industrial scenarios, as well as the academic research for combinatorial testing technique. To this end, on the basis of the research results of this group, a suite of combinatorial testing tools has been developed, whose functions include test case generation, test case optimization, and etc. For the requirements from both industrial and academic scenarios, the tools should be configurable, scalable, modular, and etc. This paper gives a brief introduction to the design and implementation of these tools. Keywords-combinatorial testing, combinatorial testing tools, test generation, test prioritization.

APPLICATION OF OPTIMAL MINING TECHNIQUE TO PILOT PROJECT
The modified OMT algorithm and its application to the pilot project are presented below:

Steps of optimal MINING technique 3.1.1. Step-1
Determine the regular and embedded system specific input variables from the test requirements specification. Input variables that are of continuous nature have been selected. In this OMT method only, the multi-input variables that are of continuous in nature have been considered. The details of input variables i.e. regular and ES specific selected are shown in Table 1 and Table 2. The details of output variables i.e. regular and ES specific selected are shown in Table 3 and Table 4. The range of values that must be used for generating the test data have also been pre-identified and mapped with input variables. The relationships that exist between the input variables and its corresponding related input variables can be used as the basis for generating the test cases.

Step-2
Determine the input-input relationships and also the relationships with the output variable which can be used as a database for mining relationship pattern for further modelling. Sample associativity and the relationships among the input and output variables are shown in the Table 5.   INIT-MESSAGE  -LCD-STAT  LCD-WRITE  -----2.  MSG-PW-ENTRY  -LCD-STAT  LCD-WRITE  -----3.

Step-3
A set of input variables occurs in union. The set of input variables behave in a pattern. The pattern of occurrence of the input variable can be determined by using a mining algorithm or through manual inspection. Following are the input sets and the pattern of occurrence of those sets which can be mined or manually determined.

Step-5
Trace out the output vectors having variables of similar nature and domain. Following are the output vectors related to example application.  Step 8: Stop ()  ISSN: 2088-8708

COMPARATIVE ANALYSIS
Three methods exist in literature which can be used for generation of test cases that can be used for testing the embedded systems. The methods include generation of test cases using input domain, output domain, and generation of test cases for semi or pseudo exhaustive testing using genetic algorithms. All these methods do not take into account interrelation ships between input variables.
The association between the variables is only limited to adjacency. The methods are compared considering the testing requirements of the embedded systems and the techniques that must be used for undertaking the testing of the embedded systems. Table 7 shows comparison based on the suitability to generate the test cases that can be used for testing different features of the embedded system. From the table it can be seen that OMT is made for testing the embedded systems considering all the features that are related to the embedded systems. Table 7. Comparison of the Test case generation methods (Combinatorial Methods) based on the suitability of the same for testing the embedded systems