Stock price forecasting in Indonesia stock exchange using deep learning: a comparative study

ABSTRACT


INTRODUCTION
Electing stock issuers for investing short-range had a high-risk failure because the stock market influences by many factors, which created an unstable stock market environment [1].The task of electing best-fit model forecasting on each stock issuer is urgently needed to reduce the failure effect [2].The Indonesia stock exchange (IDX) had attractive growth for investors.In 2022, the market share grew to 34.27%, and market capitalization reached 15.06%.Stock issuer listed on IDX is 825 companies.The stock issuer is separated into 11 sectors: raw goods, non-primary consumer goods, primary consumer goods, energy, infrastructure, health, finance, industry, property, technology, and transportation [3].The best architecture model for each sector is essential to maximizing stock returns on growth IDX.
The IDX is already used in previous research but only uses a few stock issuers.The LQ45 index is used in previous research [4], [5], and [6] because the LQ45 index builds up from 45 companies with excellent company performance and can affect composite stock price index.Weaken of the composite stock price index occurred when the price of the LQ45 index dropped and vice versa.In other research, fewer  ISSN: 2088-8708 Int J Elec & Comp Eng, Vol.14, No. 1, February 2024: 861-869 862 companies are used to forecast stock prices, especially those with significant market capitalization, such as PT Bank Central Asia Tbk (IDX:BBCA), PT Bank Mandiri (Persero) Tbk (IDX:BMRI), and PT Bank Rakyat Indonesia (Persero) Tbk (IDX:BBRI) [7], [8].The large market capitalization is chosen because stock price movements tend to climb since the initial public offering (IPO) [9].The LQ45 only has 45 companies, and the large market capitalization usually only uses the top ten companies, which indicates that the other companies in 825 stock issuers have not been tested yet.We propose using 88% of the stock issuer population to discover that the issuer has the lowest risk of failure.
Stock price forecasting already uses several models, such as statistical [10], pattern recognition [11], machine learning [12]- [14], and deep learning [15]- [18].The deep learning model using convolutional neural networks (CNN) and recurrent neural networks (RNN) outperformed statistical and machine learning models [15], [18], [19].In previous research, the combination of the CNN-long short-term memory (LSTM) layer performs better than the standalone CNN or long LSTM layer.The CNN layer successfully extracts features from the input data and passes them to the LSTM layer to predict the output [16].The LSTM layer surpasses nine machine learning models.The cell of LSTM can recognize the pattern of random time interval data [19].The GRU layer performance slightly surpasses the LSTM layer [17].Graph convolutional networks (GCN) perform better than gated recurrent units (GRU) on stock price forecasting.The GCN had a correlation matrix as input data to recognize market signals [20].Although previous studies have presented the comparative performance of the deep learning model with other models, the consensus of the best architecture is not fulfilled yet because the model architecture is not tested with a large dataset, and the alternative stacked layer model has not been implemented.
In stock price forecasting, no model or parameter can be best-fit for all stock issuers.The model and parameters may be the best option for some issuers but the worst for others.Varying architecture models are required and must be compared in performance before being used in problem-solving.The best architecture was determined by comparison using particular statistical criteria.This study tests the CNN, LSTM, GRU, GCN, and mixed layer models on each stock issuer.The best architecture is determined by voting consisting of five evaluation metrics.Each stock issuer will have a best-fit architecture to reduce stock forecasting failure.
The main contribution of this study is a comprehensive benchmark that compares the performance between CNN, LSTM, GRU, GCN, and mixed layers to fulfill the consensus best architecture on stock price forecasting in IDX.The forecasting models used 727 of 825 stock issuers in the IDX and then measured performance using mean squared error (MSE), mean absolute percentage error (MAPE), coefficient of determination (R 2 ), root mean square error (RMSE), and f1-score.The impact of this study is not only for the scientific literature but also for investors to reduce risk failure on stock price forecasting in terms of the ability to forecast the IDX.The rest of the article is organized as follows: section 2 presents the research method, section 3 results and discussion, followed by a conclusion in section 4.

METHOD
Our comparative deep learning model in stock forecasting uses four layers: CNN, LSTM, GRU, and GCN.Each layer will be stacked with a maximum of two layers.The dataset and experiment setup are presented in this section.

Dataset
The stock issuer listed on the IDX amount to 825 in 2022.Table 1 shows 727 companies spread across 11 sectors, and 2,588,451 data rows are used in this study.The selected issuer is based on criteria with a minimum of 200 days of the stock transaction to meet the needs of time-lag variations, and the variance on the data test is not zero.The dataset stock transaction consists of open price, highest price, lowest price, close price, and volume of transactions.

Model forecasting
The combination of CNN, LSTM, GRU, and GCN is used to get the best architecture on stock price forecasting for each issuer.The explanation of each layer is presented in this sub-section.The characteristics of each layer are explained to understand their contributions to the forecasting process.

Convolutional neural networks
Convolutional neural networks (CNN) are feed-forward neural networks that excel in computer vision and text processing.CNN architecture is already used to forecast time series data, especially extracting features from raw input [18].The CNN architecture uses local perception and weight sharing to enhance the efficiency of the learning model.The fully connected, pooling, and convolutional layers are stacked to build

863
CNN architecture [21].The critical layer of CNN architecture is the convolutional layer, which contains the kernel to extract data features.The convolution process calculation in the convolutional layer is shown in (1).
The pooling layer had a task to reduce the dimensionality of data which impacts the training cost networks.The last fully connected layer received flattened input from the convolutional and pooling layer to produce the final output [16].
The value of   represents the result of the convolutional process, which is formed using the ℎ activation function.The input vector   , the convolutional kernel weight   , and the convolutional kernel bias   are used in ℎ activation function.These components collectively contribute to calculate   , which reflects the converted and filtered information derived from the input.

Long short-term memory
Long short-term memory (LSTM) is designed to solve vanishing gradient limitations in standard recurrent neural networks (RNN).Three gates link memory cells in LSTM networks: forget, input, and output.All gates had the task of controlling the flow information-passed of the network [7].LSTM implementation is shown in (2) to (7).
The forget gate   is calculated using the sigmoid activation function , the input value of the current state   , and the output value from the previous cell ℎ  −

Gated recurrent unit
Another variant of RNN is GRU, which is simple than LSTM architecture.GRU architecture only has two gates to control information on neural networks.The reset gate had the task of forgetting information from the previous state.The update gate had to control how much information was added to the current state [22].GRU Implementation is shown in (8) to (11).
The forget gate   and update gate   are produced using the sigmoid activation function , input the current state   , and the previous output state ℎ −1 .The candidate current state ℎ ̃ calculated using activation function ℎ, forget gate   , the previous output state ℎ −1 and input the current state   .Final output GRU ℎ  is used update gate   , the previous output state ℎ −1 and the candidate current state ℎ ̃.The weight on each gate is represented by   ,   , and  ℎ ̃.

Graph convolutional networks
GCN address limitations in prior semi-supervised node classification, in which the edges encode only node similarity and optimize a multi-step pipeline [23].GCN used graph-structured data as input, which contains node and edge, and then used a matrix of the undirected graph in the convolution process [20].The implementation of GCN layer-wise propagation is shown in (12).
() is the matrix of activations in the  ℎ , with  (0) = .The activations matrix produced using activation function  = .The adjacency matrix of the undirected graph  ̃, and layer-specific trainable weight matrix  ̃ and  () .

Experimental setup
As shown in Figure 1, each stock issuer from Table 1 past pre-processing process to identify missing data was removed from the dataset.The dataset is split into 80% training data and 20% testing data, and the amount of data depends on the date IPO of each stock issuer.The model consistency was tested with 5, 10, 15, 20, and 25 time-lag data.The various time-lag represents the unstable environment of the stock market.The pre-processing process results in an input matrix time series consisting of number samples, time lag, and features, or an input matrix undirected graph with dimensions consisting of number samples, nodes, and nodes.This research produced the nodes representation of time-lag features and edges by correlation score.

865
The CNN, LSTM, GRU, and GCN layers from the TensorFlow (TF) framework are used to learn the input matrix and resolve stock price forecasting.Standalone layer or two layers combination is used to identify which layer significantly affected stock forecasting.As shown in Table 2, the only model using the GCN layer receives an input matrix undirected graph and input matrix time-series; the other only uses input matrix time-series.

Evaluations
This study proposes voting using five evaluation metrics to determine the best architecture stock price forecasting on the IDX.The voting has a rule of 2 of 5 best metrics to determine the best architecture on each stock issuer.The performance comparison used five evaluation metrics: the R 2 score and RMSE to determine architecture fitness with the data and the MSE and MAPE to measure error values from the forecasting model [24].The last metric used is the 1-score to measure the performance model on predicting price movement direction.The actual   and predicted  ̂ direction calculation shown in (13) and (14).The up direction is represented by 1, and the down direction is represented by 0.
The confusion matrix can be utilized to produce the 1-score.The true positive (TP), true negative (TN), false positive (FP), and false negative (FN) calculate using   and  ̂ value.The 1-score calculation shown in (15) utilized precision () and recall () based on TP, TN, FP, and FN [25].

RESULTS AND DISCUSSION
The code and the experiment results are publicly available on a GitHub repository to make the research reproducible [26].This study uses the layer CNN, GRU, LSTM, and GCN to produce 11 architectures and forecast stock prices the next day.The voting with minimum rule best 2 of 5 evaluation metrics determines outstanding architecture.The TFGRU is the best architecture that produced the best performance on 315 of 727 stock issuers.TFGRULSTM architecture is the second-best architecture, producing the best performance on 148 companies.
As shown in Figure 2, the rank four lowest achieved by models use two layers, and the first layers use GCN or CNN.The convolutional layer cannot extract essential features from input data and then pass The GCN or CNN is the best architecture for some stock issuers but not good enough for the rest companies.The top three best performances are produced by GRU or LSTM layers, meaning GRU and LSTM are the best layers to build stock price forecasting on the IDX.

Figure 2. Architecture performance comparison on stock price forecasting
The performance comparison of architecture forecasting is breakdown into four evaluation metric regressions using average scores from 727 stock issuers, presented in Table 3. Confirmed the TFGRU as outstanding architecture with MSE, RMSE, and MAPE had the lowest error.The TFGCNLSTM outperforms the others model with the highest R 2 score, but the average R 2 score on all architectures is negative, meaning all architectures do not fit almost stock issuers.The TFGCNGRU is the worst architecture using MSE, RMSE, and MAPE evaluation.TFCNNGRU achieved the lowest fitted model in the IDX.The prediction evaluation of the stock price movement direction is essential because the prediction error can inflict financial loss on investors.The actual movement direction is down, but the model predicted is up.Then the investor decides to sell the stock, and the investor will lose the deviation between the actual value and predicted value multiplied by the shares sold.
The highest f1-score in Table 3 is TFCNNGRU architecture, with a score is 0.4658.All architecture had performance under 0.5, meaning the investor has under 50% chance to take profit from selling shares if using one architecture to all stock issuers.The TFLSTM outperformed the other architecture based on accuracy performance with an accuracy is 0.5174.Based on accuracy performance, all architectures had high risk if used in the real-time stock market, and deep analysis is needed to pick the best architecture for each stock issuer.
The top ten issuers in  4, TFGCNLSTM is used by the issuer JKSW and PURE from the raw goods sector to produce the best performance.The issuer JKSW has a MAPE of 0.0003 with an accuracy of 0.97 and an f1-score of 0.78.The issuer PURE has an accuracy of 0.99 and an f1-score of 0.75 with a MAPE of 0.01.The TFGCNLSTM is the second worst on overall performance comparison, but it is still needed on some stock issuers to get the best forecasting performance.As shown in Table 4, the various architecture is used to produce the highest performance on each stock issuer, meaning each issuer needed fitted parameters and architecture to achieve the best performance.
Based on votes, performance comparison, and top ten company analysis, TFGRU architecture was determined to be the best architecture for stock forecasting in IDX.The TFGRU yielded the best results on 315 companies, with an average R2 is -40.1303,MSE is 127,978,064.19,RMSE is 553.327,MAPE is 0.858, and f1-score is 0.456.On five of the top ten stock issuers with the best model performance, the TFGRU is the best architecture.The TFGRU evolved into the ideal architecture due to its ability to extract data sequences from time series data.

CONCLUSION
This research proposed benchmarking the performance CNN, GRU, LSTM, and GCN layer to stock price forecasting on 727 companies in the IDX.Consensus on the best architecture has been fulfilled with TFGRU as the best model forecasting in the IDX.The architecture built from LSTM or GRU layer produces the finest results on performance comparison and confirms best result on previous work is influenced by LSTM or GRU layer.The various time lags and different architectures are used on the top ten stock issuers with the best forecast performance, meaning fitted parameters and architecture are needed on each stock issuer to produce optimum model performance.This comparative research can still be refined to boost existing performance.The hybrid forecasting and classification model can be developed to increase the performance of predicting stock price movement direction without lowering the performance of regression forecasting.
Int J Elec & Comp Eng ISSN: 2088-8708  Stock price forecasting in Indonesia stock exchange using … (Agus Tri Haryono)

Table 1 .
Dataset of stock issuer grouped by sector 1 .The forget gate weight is   and  ℎ .The forget bias represented by   .The input gate   is determined using the sigmoid activation function , the current state input value   , and the preceding cell output value ℎ −1 .  and  ℎ are the input gate weights.The input gate bias is represented by   .The output gate   is calculated by combining the sigmoid activation function , the current state input value   , and the preceding cell output value ℎ −1 .  and  ℎ are the output gate weights.The output gate bias is represented by   .The candidate current state ̃ is calculated using the activation function ℎ, the current state input value   , and the output value ℎ −1 of the preceding cell.The candidate current state weights are   and  ℎ .  represents the forget gate bias.The current state   is updated using forget gate   , previous state  −1 , input gate   , and the candidate current state ̃.The current state   and output gate   are used to produce the final output ℎ  .

Table 3 .
Architecture score evaluation comparison on stock price forecasting Table 4 are elected by sorting the highest f1-score, highest accuracy, lowest MAPE, lowest RMSE, lowest MSE, and highest R2 .The TFGRU architecture still dominates to produce the 867 best performance on issuer SRIL, ARKA, PURA, LCKM, and SOFA.The highest f1-score achieved by SRIL with a score is 0.81 and accuracy is 0.99.The SRIL issuer used a time lag of 10 and TFGRU architecture to produce low risk for investors because it has high accuracy and f1-score on prediction stock price movement and a low MAPE is 0.03.The second-best issuer is NANO, with a time lag feature of 5, and TFCNNGRU architecture produced an f1-score of 0.80, an accuracy score of 0.90, and an MSE score of 0.92.In Table3, TFGCNLSTM is the second worst architecture by MSE, RMSE, and MAPE evaluation, but in Table Stock price forecasting in Indonesia stock exchange using … (Agus TriHaryono)

Table 4 .
Best-fit parameter on top ten issuers with best model performance