Empirical evaluation of continuous auditing system use: a systematic review

For more than two decades, the concept of continuous auditing (CA) has been introduced and many large firms had taken the initiatives to apply the CA system in supporting their audit functions. Despite the benefits that the CA system is offers, it is still not widely use and the number of people using the CA system is still considered low. This research focuses on the published papers on the use of CA system within the context of auditing system addressing the quality of system implementation. This paper analyzed primary studies collected using the pre-determined search strings on nine online databases. As a result, a total of 60 articles were carefully selected to undergo further analysis based on empirical evidence of CA system use. The articles were analyzed qualitatively using ATLAS.ti 7 and the elements for the CA system use are extracted from the selected papers. A total of four elements were identified contributing to the use of CA in practice. Those elements are the participant quality, system quality, information quality and products and services quality. This study answers five research objectives to understand the current studies on CA and to determine future research works on CA.


INTRODUCTION
The computerization of accounting operations in the early 1970s has led to the development of electronic data processing (EDP) which also promote the evolution of continuous auditing (CA) simultaneously [1]. CA has been introduced by Vasarhelyi and Halper since early 1990's through the introduction of CA model which is known as continuous process auditing system (CPAS) to facilitate the bills processing at the large data centers [2]. Vasarhelyi and Halper [3] mentioned in their study that due to the introduction of the system that provided the auditing of large paperless real-time system, it had aroused the users to think about the alternative to traditional auditing [4], [5]. Many follow-up studies regarding CA in the following years have been inspired by this model [2]. CA is highly demanded nowadays due to the introduction of section 404 of the US Sarbanes-Oxley Act by the US government which highlighted the To address the research questions, researchers identified the definition of continuous auditing by using ATLAS.ti 7 as the qualitative data analysis tool. ATLAS.ti 7 helps researchers analyze 60 articles that were selected to compare and construct the definition regarding continuous auditing versus traditional auditing. The above questions would enable researchers to have a better understanding on the current state of the continuous auditing system use within internal auditors. Table 1 shows the five elements known as population, intervention, comparison, outcomes and context (PICOC) necessary for the formulation of research questions on effectiveness as suggested by [27]. The population refers to the community of interest in the study; intervention is the focus of the review (i.e., investigating continuous auditing) while comparison refers to the alternative of auditing process we would like to compare with. Outcomes in PICOC refer to the measures used in the primary studies, and context refers to the context of the primary studies. Table 1. Summary of PICOC for continuous auditing studies [27] No Elements Definition 1 Population Internal auditors 2 Intervention Continuous auditing 3 Comparison Traditional auditing 4 Outcomes Elements for the CA system use 5 Context Within the domain of auditing system addressing the quality of system implementation

Systematic review process
The review process of the literature was performed according to several steps as suggested by [27] in Figure 1. The steps include, formulation of research questions, identification of research keywords, identification of relevant literature, selection of primary studies based on the inclusion and exclusion criteria, assessment of the studies quality based on several characteristics and interpretation of the results. The final part involves the synthesis of evidence where the results from primary studies are combined and interpreted. In the next sections all the steps are detailed from identifying the sources of information in online databases to paper selections and evaluation until the reporting of the findings of the SLR process.

Sources of information
As an initiative in gaining a broad perspective on the current practice of continuous auditing within the internal auditors, several online databases subscribed to by the Universiti Teknologi Malaysia's library were utilized to collect an extensive set of published continuous auditing academic literature. The following list shows the databases that have been covered: − ACM Digital Library (<portal.acm.org>) − American Accounting Association(<aaajournals.org/>) The above databases indexed the most relevant journals, conferences, and workshop proceedings available within the software domain. The selection of online databases was based on our knowledge of databases that indexed studies related to internal auditors, continuous auditing, traditional auditing, real-time reporting, and quality studies. This study also searched for keywords "internal auditor" or "internal audit" or "continuous auditing" or "continuous audit" or "traditional auditing" or "traditional audit" or "real-time reporting" or "real-time reporting" or "quality" or "excellence" using web search engine run by Google (Google scholar) to search for the full text of related articles.

Search criteria
SLR approach provides the sensitivity and specificity of the search which is crucial in obtaining the specific search results. The sensitivity of SLR is related to a search that is able to retrieve a higher number of relevant studies while the specificity of SLR enables the search to retrieve a minimum number of irrelevant studies [27]. The major terms that were used in the review questions were identified based on the PICOC. Additionally, the following strategies were used to construct the search string: i) list down all the keywords that were mentioned in the articles, ii) search for the synonyms or alternative words of each keyword, iii) use the Boolean OR to incorporate alternative spellings and synonyms, and iv) use the Boolean AND to link major terms from population, intervention, and outcome. As a result of the above strategies, the complete search string used for the search of online literature is (internal auditor OR internal audit) AND (continuous auditing OR continuous audit) AND (traditional auditing OR traditional audit) AND (real-time reporting OR real time reporting) AND (quality OR excellence).

Data extraction and quality evaluation
Regarding the data extraction, a form was designed and applied to gather the evidence related to the research questions. The quality checklist for primary studies was designed based on previous research [28]. The checklist was modified according to the research question and used to measure the quality of the primary studies. Seven general questions were used to measure the quality of both quantitative and qualitative studies according to the following scale: Yes=1 point, No=0 point, and partially=0.5 point as shown in Table 2. The total quality score for each study ranged between 0 (very poor) and 7 (very good). The papers that obtained very poor scores are excluded.

Study selection
A set of inclusion and exclusion criteria was defined for the selection process from the literature that have been identified in the databases. These criteria are used to guide the selection of relevant publications for further analyses. In the preliminary selection, publications based on empirical evidence on CA use were selected thus eliminating publications based on reviews or describing tools (software or hardware). In the primary phase, titles, abstracts, and data collection methods are used for the selection of the articles. In addition, for a thorough analysis, only full research papers were included in the analysis, thereby excluding posters, abstracts, and short papers. Retracted articles and articles unavailable in English were also excluded from the study.

Limitations
To fit our purpose of study, a time frame from January 2000-September/October 2017 was established for the selection of studies as the main objective was to provide a better analysis of published reports set on continuous auditing practice. Since the term continuous auditing was introduced in the late 1990s, a selection of articles published from January 2000-September/October 2017 would be specific to continuous auditing. The selected literature were cautiously and carefully reviewed to ensure that the manuscripts were not duplicated, and only primary studies were selected. Besides, the papers were selected through keywords searching and analysis of references. After reading the title, abstract, and conclusions, the papers had undergone the exclusion process. However, due to the application of searching and exclusion method, there is possibility that some papers which are considered suitable with the research were excluded and missed.

RESULTS AND DISCUSSION
This section presents the synthesis of evidence of our SLR, beginning with the analysis of the literature search results. In our preliminary search, we retrieved a very small number of articles when using the complete search string defined in the protocol. Based on the search string identified, the number of articles retrieved from online databases is shown in Table 3. A total of 60 articles were selected as primary studies using inclusion and exclusion criteria as mentioned in section 2.5. The Emerald database papers comprise 50% of the relevance and evidence-based study. A quality checklist comprising seven general questions is intended to measure the quality of quantitative and qualitative studies using an ordinal scale from 1 (very poor) to 7 (Excellent). These questions were designed based on guidelines reported in Okoli and Schabram [28]. In the process all the papers were evaluated, and the scores are marked. The quality assessment indicated that 42 papers (70.00%) were of excellent quality, 8% considered good and 22% scored fair in evaluation as depicted in Table 4. The extract of quality evaluation performed on the 60 selected papers is shown in Figure 2.   [29], they had combined Ritchie [30] three levels of information along with Mason's expansion of effectiveness or influence level [31] to generate six categories of information systems which are known as information quality, system quality, individual impact, user satisfaction, use, and organizational impact. After ten years, DeLone and McLean [22] had introduced the new version of D&M IS success model which includes system quality, information quality, service quality, user satisfaction, intention to use/use and net benefits. Besides, Venkatesh et al. [23] have addressed the participant quality as one of the important elements in CA system use in the unified theory of acceptance and use of technology (UTAUT). Thus, from the compilation of several research made by previous researchers, we identified that the most relevant elements of CA system use are made up of four elements which are i) participant quality, ii) information quality, iii) product and services quality and iv) system quality in Table 5. From the 60 primary studies selected, it shows that each element was measured differently by different authors.
The participant quality also refers to the auditor's quality as it is necessary for the auditor to be competent and possess technical skills to ensure the success of CA system use as the uses of information system involves both people and technology [32]. Besides, sufficient and effective training can help in developing skilled auditor in using CA system [16], [32]. System quality refers to the assessment of the specific information processing system [29]. According to DeLone and McLean [22], from the perspective of Internet environment, system quality represents the characteristics desired when using the e-commerce system. Besides, they mentioned that system quality can be determined by the six scales which are system flexibility, system integration, response time, language, error recovery and convenience of access. The system quality can also be accessed based on the system's adaptability, reliability, availability, usability and response time [22]. Information quality refers to the importance of the information to have high reliability, reusability, availability of data and control and accessibility of transaction data [33]. Amin and Mohamed [2] stated that information quality with high reliability can provide information that is free from bias and material error to the users. Products and services quality according DeLone and McLean [22] is the support provided by the service provider to various departments or users which is considered the most important part as poor quality of user support will result in losing customers and declining in sales. Meanwhile, Chou and Chou and Chang [33] stated that product and service quality could be measured by its capability to discover and report any material misstatement in the accounting information.

Discussion
The 60 articles selected in this study are related to the CA system and obtained and selected from nine (9) databases. The relevant answers to each research question mentioned previously will be discussed in this section. These include comparison between continuous auditing and traditional auditing, the motivation to use continuous auditing and the elements determining the use of continuous auditing.

How researchers define and compare continuous auditing verses traditional auditing?
Vasarhelyi et al. [17] mentioned that traditional audit will provide assurance on the financial reports provided by the management by using manual processes and IT audit separately. Comparatively, CA provides a paperless system, real-time reporting, applying the analytics approach on large system and making a comparison between the data and acceptable standards to report any abnormality [17]. Chou and Chang [33] mentioned that in providing a web-releasing assurance, CA would be more appropriate than traditional auditing as the traditional auditing required the auditor to update and recompile the whole program constantly which made it more complicated for the auditors in identifying the changes that have been made previously. Referring to the study performed by El-Masry and Reck [34], the differences between CA and traditional auditing can be seen in the duration for the reports to be released, firm's internal control, and frequency of substantive tests performed on the transactions. Compared to traditional auditing, CA can disclose the reports within a short amount of time [34]. Traditional audits only provide a year-end audit and traditional interim [17]. Besides, during CA application, the firm's internal control will be monitored frequently to enable thorough understanding on the systems. The improvement in internal control will improve the reliability of the reports generated and disclosed which will also escalate the investors' confidence on the data provided [34]. The other difference between traditional audit and CA according to El-Masry and Reck [34] is that CA will enable the auditors to increase the frequency of substantive tests performed on some transactions which usually performed once a year if the auditor used traditional auditing. However, using CA, the tests can be performed either weekly or daily and the test can provide more evidence which will lower the audit risk and boost the investors' confidence in the information audited by the auditors [34]. El-Masry and Reck [34] Perception, Confident Relevance Assurance, Reliability S14 Esteves [39] Flexibility Cost S15 Ettredge et al. [40] Performance Access Time Efficiency S16 Ezat and El-Masry [41] Management, Independent Time S17 Tafri et al. [42] On-going, Management Relevance S18 Gonzalez et al. [43] Ease Reliability S19 Gonzalez et al. [

What motivate individuals to use continuous auditing system?
In the last two decades, a considerable amount of literature has been accumulated addressing CA. The evolvement of the idea of CA can be said to be related to the development of electronic data processing (EDP) [1], a prototype system developed by Cash et al. [85] which led to the development of a new auditing field called EDP auditing. All aims was focused on improving audit efficiency especially with the widespread use of new technologies, together with the development of the internet, the growth of electronic commerce, e-government, online securities trading, and e-procurement systems [1], [13]. Moreover, greater documentation and audit trails on the audit process were also provided by CA [8]. According to Rikhardsson and Dull [73], the use of CA system can improve the quality of data provided as the CA system can monitor the transaction recording to ensure its accuracy and protect the integrity of the systems and modules data flows. It might also be possible that the individuals are interested in using CA system as an effort in improving the internal control system and maintaining the investors' interest and confidence simultaneously [34]. Besides, the increase in financial accounting frauds and scandals such as Enron, Tyco and WorldCom cases might motivate the auditors to use CA system [38]. The continuous auditing (CA) system can help in providing higher assurance on the reports generated which can increase the investors' confidence on the reports simultaneously [21]. The study performed by Appelbaum et al. [36] mentioned that the CA adoption will help in improving several areas such as reducing the accounting error, providing more real-time data analytics, providing a triggers on anomalies for both real-time or close to real-time, and improving the effectiveness and efficiency of auditing process. Besides, according to Davidson et al. [8], CA provides improvement in the audit documentation and audit trail of the process in which can give higher audit integrity.

What elements determine the continuous auditing system use?
Our synthesis of the 60 articles shows that the successful use of CA system can be measured through four elements which are participant quality, system quality, information quality and products and services quality. In this study, the participant quality has also been addressed as one of the important elements in CA system use since the unified theory of acceptance and use of technology (UTAUT) developed by Venkatesh et al. [23] has added the participants compared to the D&M IS success model proposed by DeLone and McLean [22]. The success of CA is closely related to participant quality in which the participants are the internal and external auditors. The implementation of CA system is more significant to internal auditing environment as compared to external auditing. The internal and external auditors are required to possess sufficient audit technology to make it easier for them in detecting and reporting all possible frauds, errors and irregularities [33]. Participants involved in CA should have appropriate knowledge of respective parts of CA to avoid misunderstanding which might affect the uses of CA. It can be proven from the study performed by Davidson et al. [8] where he mentioned that it is crucial for the participant to have a sufficient understanding of the business processes and control risks of the firm along with the analytic auditing technologies during the CA system implementation to guarantee that all possible red flags are highlighted. Besides, as a requirement in using CA, Amin and Mohamed [2] added that it is crucial for the auditor to be given permission to access the client's system to enable the accumulation of required evidence using the tools available in the client's system. The next element is related to the system quality which concerns more on terms such as easy to use, the convenience of access, ease of learning, flexible, reliable, accurate and meeting user requirements [22]. According to Esteves [39], the system must be easy to understand and can provide better information quickly to users and can reduce the cost to provide higher efficiency. Besides, the system must be quick in providing the information needed without any lagging process as it will not disrupt the audit efficiency. Gonzalez et al. [43] added that the CA system is convenient to learn and its implementation can minimize the number of work which is fundamental in performing audit works. CA is an important application in providing assurance and monitoring business and financial information [2]. In case of larger data or samples, it is more costeffective to implement the CA system as it enables the auditors to test and analyze the large data quickly and effectively compared to traditional auditing [16], [63]. In addition, Rosa and Caserio [74] stated that the technology available within CA can minimize the total hours required by the external auditors to perform risk evaluation, obtain sufficient knowledge regarding the processes, and provide a reasonable assurance that the internal controls of the firm work as planned.
Information quality is also considered the most important element in CA system application. This is because inaccurate information or low information quality such as data duplication, manually recorded data entries, centralize databases, information accessibility problems, and poor communication systems can affect the process of decision making [32]. Chou and Chang [33] mentioned that the reliability of the information can be obtained through the third-party which is also a competent auditing service. According to Amin and Mohamed [2], the information's reliability, relevance and comparability can be improved through the use of CA in which the financial information that has been reported is more understandable. The reliability of information refers to the information being free from bias judgment and material error and can be used confidently by the users in making decisions and transactions [2]. Amin and Mohamed [2] also mentioned that the relevance of information refers to how relevant the information which can affect the process of decision making. Besides, the auditors can perform the comparability of information using CA system to increase the assurance of the information quality that has been disclosed and reduce the data manipulation risk by the company [2].
Products and services quality also plays a main role in affecting the uses of CA system. The products and services quality refers to the support provided to the users upon using the tools or in this case, the CA system [22]. Without proper support given to the users, users might be dissatisfied and in the worst case scenario, results in the abandonment of the CA system itself [22]. The audit quality can be accessed through its proficiency in detecting and reporting any material misstatement discovered in the accounting information [33]. Meanwhile, Bierstaker et al. [38] added that audit quality can be measured by the audit procedure's efficiency in detecting and reporting fraud that might occur or available in the information. Rosa and Caserio [74] stated that from CA system used, the amount of errors can be minimized, and it can lower the costs and reduce the potential risk in future losses of revenue. Furthermore, it can also improve the customers' satisfaction with the company's services as errors and irrelevant service offers are not being made [74]. Moreover, Davidson et al. [8] mentioned that the timely verification made by CA system on the integrity of transactions can improve fraud prevention and reduce the pressure made by the management on the internal auditors simultaneously.

CONCLUSION
This paper presented a systematic review of the CA empirical studies aimed to investigate the current use of CA system in the context of internal auditors of private organizations and the elements that determine the CA system use. The systematic literature review on CA was addressed through the inclusion and exclusion criteria, the research questions and the search keywords used. All 60 primary studies were selected from journals within nine (9) online databases and 42 papers (70%) were of excellent quality. Moreover, only papers containing data were selected. The results of this study were classified into several categories and analyzed. The research areas within the CA system use were presented in this paper by answering the research questions that have been defined initially. This paper also presented the results from the systematic literature review on the empirical evaluations of CA system use. Based on our research questions, we have discovered that: there are three (3) studies that discussed on the differences between CA and traditional auditing in details, six (6) studies mentioned on the things that motivate individuals to use CA system there are four (4) elements that determine the CA system use.
We have identified a lack of focus on the participant or user rather than the technology itself from the study. Thus, we have combined the D&M IS success model from DeLone and McLean and UTAUT. The D&M IS success model does not particularly mention the participant quality as one of the elements in CA, but UTAUT had placed an important role on the participants in the implementation of CA. Thus, we had included the participant quality as one of the important elements along with the system quality, information quality and products and services quality resulting to four (4) elements that determine the CA system use. This paper focuses on internal auditors as a future work, research may also be focused on the external auditors' adoption of the CA system. Further research can also consider focusing on the mediator's role as a determinant of the CA system use.