Data modelling and data processing generated by human eye movements

Received Oct 6, 2020 Revised Mar 6, 2021 Accepted Mar 19, 2021 Data modeling and data processing are important activities in any scientific research. This research focuses on the modeling of data and processing of data generated by a saccadometer. The approach used is based on the relational data model, but the processing and storage of the data is done with client datasets. The experiments were performed with 26 randomly selected files from a total of 264 experimental sessions. The data from each experimental session was stored in three different formats, respectively text, binary and extensible markup language (XML) based. The results showed that the text format and the binary format were the most compact. Several actions related to data processing were analyzed. Based on the results obtained, it was found that the two fastest actions are respectively loading data from a binary file and storing data into a binary file. In contrast, the two slowest actions were storing the data in XML format and loading the data from a text file, respectively. Also, one of the time-consuming operations turned out to be the conversion of data from text format to binary format. Moreover, the time required to perform this action does not depend in proportion on the number of records processed.


INTRODUCTION
One of the important tasks in research is the processing of large data sets that are generated in various experiments. The many possibilities for generating experimental data and their heterogeneous nature lead to certain difficulties in their modeling. Our aim is to model the data so that the result set is easy to analyze. In the present study, we modeled the experimental data according to the relational model [1]- [4], but used client datasets [5] to store, process, and analyze the information. Thus, the summarized experimental data were convenient for analysis, which allowed to formulate the relevant conclusions. After considering the importance of this aspect of research in order to easily share the results obtained, we were motivated to present the research work in this article.
Many approaches to creating visual tasks based on glass patterns (stimuli) are considered in various scientific studies [6], [7]. In recent years, various areas that are related to the analysis of human decisionmaking processes have been explored [8]- [11]. Techniques in the field of artificial intelligence are widely used [12]. They are used to model experimental processes [13]- [15], to analyze experimental results [16], and to examine a variety of indicators, such as age and/or gender [17], [18]. One important problem associated with this type of research is the processing of the data used. Different data models are used, for example relational and object-oriented [5], [19], [20], different data access technologies, for example, remote database servers and web services [21], and different software approaches [22], [23].
The subject of the present study is an approach to processing heterogeneous information generated by different devices and different software applications. The data is processed through client datasets and can be stored in XML files or in binary files (used by the client dataset). The advantage of using the xml format is that it is portable and self-describing [24]. The disadvantage of the XML format is the multiple increase in file size due to the addition of self-descriptive tags to the data [25]. When storing data in the binary file format used by client datasets, the file sizes are significantly smaller. This allows faster buffering of data from files in the computer's memory. In addition, the client datasets provide methods for converting data from the internal (binary) format to external text formats (for example, as XML or CSV).
The file converter application was developed for the purposes of this study. It provides the ability to process different types of files related to experiments. Such files are, for example, the stimulus files (BIN), the files with information about the movement of the eyes during the experiments (EYE), the result files from the experimental sessions (DAT) and others. Other features of this application are also loading, processing, filtering, converting and exporting all supported files. Thus, this application is actually a conversion module. This provides the ability to process heterogeneous information (collected from different files (with different file structure)), which in the next step is converted to a common file format (with binary structure). Once the application has converted the heterogeneous information to a common format, it can merge, split, reorganize the data at the structural level and export this data to external files (with a different file structure if necessary). Note that when the data is finally exported, it is already merged into a common file format, unlike its original form, which is heterogeneous. Different programming languages and methods for software development and testing are presented in [26]- [28].
In order to provide access to all collected information (including visual stimulus and result files), it is necessary to choose an approach to modeling the data [29]. The aim is to easily summarize and analyze these data. Two approaches are most commonly used for this purpose [5]. The first approach is to store the data as it is (raw) and then, just before analyzing it, to summarize it. The second approach is that the data is imported into a pre-designed local or remote database, from which the summary information is then extracted for analysis via SQL queries [30]. Access to such a database can be done through a database management system-DBMS [1], [31]. Architectures of multi-user systems based on remote database servers with the ability to perform calculations and analysis of experimental data have been presented in other research [21], [30].

RESEARCH METHOD
The experimental data that need to be processed are generated by different applications during the experimental sessions. Therefore, the purpose of this article is not to present an analysis of the experimental results, but to present a way to process the generated data during these experiments. Visual tasks (stimuli) are visualized on a computer display for all participants in the experiment. The stimuli are preliminarily generated, with each stimulus containing exactly 100 frames. In turn, each frame contains exactly 50 points. The rules for generating stimuli are described in detail in [32].
During each experimental session, 140 stimuli were visualized sequentially for each participant in the experiment. The order in which the stimuli were visualized was predefined. During each experimental session, the eye movements of each participant were recorded. This is done with a special device-Saccadometer and specialized software connected to it. The result files contain the information about the eye movements of each participant for each experimental session. These files have a specific file format that is not officially published. Therefore, only specialized software associated with the saccadometer can process these files. However, the information from these files can be exported to external text files (with a .TXT extension, respectively). Table 1 shows the structure of the exported data (as text).
The exported files are characterized by the fact that they are large in size (in the order of tens of megabytes). This is because the eye movements of the participants were scanned at a sampling frequency of 1000 Hz (i.e. 1 kHz). This means that 1000 sets of values were generated in 1 second from the saccadometer. Since each stimulus contains exactly 100 frames, and each frame has a maximum duration of 33 milliseconds, the total time to visualize a stimulus is approximately 3300 milliseconds. After each stimulus, there was a pause of 500 milliseconds before the next stimulus was visualized. At the beginning of each experimental session, the saccadometer was calibrated according to the position of the participant's head. The time for this calibration is approximately 900 milliseconds. On the other hand, each participant could (via some of the mouse buttons) stop prematurely visualizing each stimulus. This usually happened when a participant "recognized" the direction of the stimulus before all the 100 frames of that stimulus were 4347 visualized (i.e., before 3300 milliseconds had passed). Given all these specific features and variable parameters of the experimental sessions determine their different duration. Therefore, in some experimental sessions, the saccadometer can generate up to over 800 000 sets of values. These "sets" of values include all parameters (fields), which were presented in Table 1, respectively -"No", "Eye X", "Eye Y", "Screen", and "Mic". The angle (in degrees) at which the participant's view by the X-axis is shifted. 4 Eye_Y The angle (in degrees) at which the participant's view by the Y-axis is shifted.

Screen
Presence or not of stimulus on the screen. Values above 800 indicate that a stimulus has been displayed on the computer screen at this time. 5 Mic The value of the microphone signal at the same time (in decibels).
Storing data in text format does not provide a convenient way to process it. These data need to be converted to the relevant base (structural) types. The conversion of data (from text format) can be done before the process of their export (to files with a specific file structure depending on the type of the respective values). Initially, in this process, the data is buffered in a two-dimensional string array (matrix). This data structure has a dimension of 5 columns (as many as the number of exported fields from the text files) and a number of rows as many as the number of generated sets with values from the saccadometer. Since the buffered data are of a character type (string), in the next step it is necessary to convert it to the corresponding base types. In order to implement this step of the data conversion process, a relational scheme "EyeMovments" was created with the following analytical presentation: EyeMovments = {No, EyeX, EyeY, Screen, Mic}. The attributes of this relational schema correspond to the fields of the text file, but their internal representation is of the corresponding base type (domains) depending on the nature of the data. Table 2 presents the structure of the "EyeMovments" relation. The "Class name" column in Table 2 contains the names of the corresponding classes (of the corresponding attributes) used by the client dataset. The primary key of the "EyeMovment" relation is the attribute "No". The values of this attribute are unique because they correspond to specific milliseconds for a given experimental session. Therefore, the values of this attribute also guarantee the uniqueness of each row in the "EyeMovments" table. The other four attributes -"EyeX", "EyeY", "Screen" and "Mic" have similar meanings as the fields described in Table 1.
The need for automated data processing led to the development of the file converter application. This application has functions for loading, processing and storing data (respectively files) used in the experimental sessions. The file converter application converts the data from strings to the corresponding base types (for example in integer, word, and double). The converted data is buffered in client datasets. Therefore, we will use the abbreviation CDS (short for Client DataSet) to denote them.

RESULTS AND DISCUSSION
The subject of this study is the file converter application and its main functions. During the experiments, the application was run on a 64-bit Windows 10 operating system. The hardware configuration has the following characteristics: Intel (R) Core (TM) i5-8265U processor at 1.60-3.90 GHz and 8 GB DDR4 at 2400 MHz random access memory (RAM). Among 264 experimental sessions (containing a total of 38 259 visualized stimuli of 35 participants), 10% were selected (randomly). This formed a representative sample of 26 experimental sessions. The data from all analyzed experimental sessions were initially stored in text files (generated by the software distributed with the saccadometer). Through the file converter application, these text files were converted and stored both in the internal format used by the client datasets-CDS, as well as in XML files. Table 3 presents the sizes of these files in MB and bytes, respectively. The last row in Table 3 contains the average values of the corresponding columns. The ratio between the sizes of the different files (depending on their format) are shown in Figure 1 (for all analyzed files) and in Figure 2 depending on the average values of their sizes (in megabytes (MB) on the x-axis).  The values in Table 3 and the diagrams in Figures 1 and 2 show that the text files have the smallest size. In contrast, XML files have the largest size. The size of the text files after conversion to XML format increases to 4.25 times. However, after converting text files to CDS format, their size increases only up to 1.45 times. In addition, the average aspect ratio of a CDS file size to an XML file size is also very large, approximately three times (exactly 2.93 times). Therefore, the CDS format should be preferred, as the sizes of the CDS files and the TXT files are commensurate. On the other hand, the data in text files are of the character type (although they "look like" numeric values), while the data in CDS format are of the corresponding numeric types. The actions that were performed by the File Converter application and that were analyzed are as follows: time to load a TXT file, time to load a CDS file, time to load an XML file, time to convert data from text to binary CDS format, time to store the data in a CDS file and time to store the data in an XML file. Due to the multitasking mode of the Windows operating system, in order to more accurately measure the average execution time of each action, five different executions of each action were made for each file. Table 4 presents the averages of the different time measurements for each action. Note that the values in parentheses indicate the average time for an action if it were to be executed with 100 000 records (rows). These values are necessary because the analyzed files have different durations. In order to make an analysis of the times for performing the different actions, it is necessary to correlate these times to a single duration (in this case 100 000 records). The last row of Table 4 shows the average values for all columns. Figure 3 shows a chart of the average execution time for loading different types of files-CDS, XML, and TXT. Figure 4 shows a chart of the average times to perform all actions (in milliseconds).  Table 4 and Figure 3 show that the file converter application loads TXT files the slowest. Loading the XML files (although they are largest) takes less time than loading the TXT files. The application loads CDS files the fastest, and compared to loading TXT files this process takes about 13 times shorter time. Compared to loading TXT files, this process takes about 13 times fewer, respectively, compared to loading XML files, this process takes about 7 times shorter time. Figure 4 shows that the two fastest actions are respectively when loading the data from CDS files and when storing the data in CDS files. Given that CDS files are also relatively small in size, it can be concluded that they are the most suitable for use, both in terms of their size and in terms of speed in processing their data. Figure 4 also shows that one of the timeconsuming operations is the conversion of the data from text format to binary CDS format (used by the client datasets). A more in-depth analysis of the values in Table 4 ("Convert (in ms)" column) shows that this action does not depend on the number of records processed. Figure 5 presents this trend. Figure 5. Influence of the number of records on the time for converting text data to binary format The indices (id) of the analyzed files are located on the x-axis of the diagram in Figure 5. The files are arranged in descending order of size, respectively 4, 3, 25, ..., 14, 13 (the right y-axis (number of records in thousands)). On the left y-axis values in milliseconds, corresponding to the times for converting 100 000 records from text format to CDS format are visualized. The two graphs in the diagram in Figure 5 are synchronized, i.e., for the same file both the number of all records and the time required for conversion 100 000 of records are shown. Figure 5 shows that the time to convert the data from text format to CDS format does not depend only on the number of records in the files. For example, it can be seen that for a file with id = 4, the time required to convert 100 000 records (from text format to CDS format) is 957 milliseconds. However, for a file with id = 3, the conversion time per 100 000 records is only 416 milliseconds, approximately twice shorter. The situation is similar for files with numbers 26 and 25, where the conversion times per 100 000 records are 886 milliseconds (for a file with id = 26) and 412 milliseconds (for a file with id = 25), respectively. These differences in values are obtained by converting the data from text format to CDS format after performing a recognition algorithm for the beginning and end of each stimulus. The algorithm analyzes the stored data related to the eye movements of the respective participant in the experiment. During the data analysis, the recognition algorithm takes into account three different factors, respectively: decision time by the respective participant (for the respective stimulus) and the values of the microphone and "Screen" signal.

CONCLUSION
One of the important tasks in research is the processing of data that is generated during various experiments. The large number of possibilities for generating experimental data and the heterogeneous nature of these data lead to certain difficulties in their modeling and processing. The present study was related to the modeling of data and analysis of the time for their processing by performing various actions. The data were generated by a special device-saccadometer and were processed through the application developed for this purpose -File Converter. An approach based on the relational model for organizing data was used, but the storage and processing of these data was done through client datasets. In this way, the summarized data were convenient for analysis, which allowed to draw the appropriate conclusions. The experiments were performed with 26 randomly selected files from a total of 264 experimental sessions. This represents 10% of all available experimental files. The data from each experimental session was stored in three different formats, respectively text, binary (internal format supported by client datasets) and XML based. The results showed that in terms of file sizes, the text format is the most compact. The binary CDS format is comparable to the text format, while the XML-based format is approximately 3 to 4 times larger than the other two formats (when storing the same data). Another important feature of the formats is that in the text format the data are of a character type, while in CDS form the data are of the corresponding numeric types. Several things were identified in relation to the data actions performed. First, the two fastest actions are respectively when the File Converter application loads the data from CDS files and when storing the data in CDS files. Second, the two slowest actions are storing the data in XML format and loading the data from a text file, respectively. Third, one of the time-consuming operations is the conversion of data from text format to binary CDS format. Moreover, the time required to perform this action does not depend in proportion on the number of records processed. This is because during the conversion of data from text format to binary CDS format, an algorithm for recognizing the beginning and end of each stimulus is executed in parallel. The execution time of this algorithm is influenced by various factors, such as the decision time of a participant (for a given stimulus), the values of the microphone signal and the values of the "Screen" signal. A direction for expanding the research is the design and development of software for recreating the experimental sessions. This specialized software must be able to process and visualize both the stimuli and the eye movements of the participants on the computer screen.