Design of Experiments vs. TOPSIS to Select Hyperparameters of Neural Attention Models in Time Series Prediction
Abstract
Attention models are used in neural machine translation to overcome the challenges of classical encoder-decoder models. In the present research, design of experiments and TOPSIS methods are used to select hyperparameters of a neural attention model for time series prediction. The configurations selected by both methods are compared with out-of-sample data in time interval between January 2020 and April 2020 when global economies were significantly impacted due to Covid-19 pandemic. Results demonstrated that both selection methods outperformed each other in terms of different output features. On the other hand, our results with more than 95 % coefficient of determination and less than 0.23 % MAPE verified that neural attention models had strong capabilities in exchange rate prediction even in extraordinary situations in global economies.
Keywords: |
Design of experiments; hyperparameter; neural attention; time series; TOPSIS
|
Full Text: |
References
K. Cho, B. v. Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk and Y. Bengio, "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation," arXiv, 2014. https://doi.org/10.3115/v1/D14-1179
D. Bahdanau, K. Cho and Y. Bengio, "Neural Machine Translation by Jointly Learning to Align and Translate," in International Conference on Learning Representations, San Diego, 2015.
R. Adcock and N. Gradojevic, "Non-fundamental, non-parametric Bitcoin forecasting," Physica A: Statistical Mechanics and its Applications, vol. 531, Article no. 121727, 2019. https://doi.org/10.1016/j.physa.2019.121727
J. Henríquez and W. Kristjanpoller, "A combined Independent Component Analysis–Neural Network model for forecasting exchange rate variation," Applied Soft Computing Journal, vol. 83, no. 105654, 2019. https://doi.org/10.1016/j.asoc.2019.105654
Z. Berradi and M. Lazaar, "Integration of Principal Component Analysis and Recurrent Neural Network to Forecast the Stock Price of Casablanca Stock Exchange," Procedia Computer Science, vol. 148, pp. 55–61, 2019. https://doi.org/10.1016/j.procs.2019.01.008
D. C. Mallqui and R. A. Fernandes, "Predicting the direction, maximum, minimum and closing prices of daily Bitcoin exchange rate using machine learning techniques," Applied Soft Computing Journal, vol. 75, pp. 596–606, 2019. https://doi.org/10.1016/j.asoc.2018.11.038
H. Y. Kim and C. H. Won, "Forecasting the volatility of stock price index: A hybrid model integrating LSTM with multiple GARCH-type models," Expert Systems With Applications, vol. 103, pp. 25–37, 2018. https://doi.org/10.1016/j.eswa.2018.03.002
Y. Liu, C. Yang, K. Huang and W. Gui, "Non-ferrous metals price forecasting based on variational mode decomposition and LSTM network," Knowledge-Based Systems, vol. 188, art. no. 105006, 2019. https://doi.org/10.1016/j.knosys.2019.105006
J. L. Minqi Jiang and C. L. Lu Zhang, "An improved Stacking framework for stock index prediction by leveraging tree-based ensemble models and deep learning algorithms," Physica A: Statistical Mechanics and its Applications, vol. 541, no. 122272, pp. 1–16, 2020. https://doi.org/10.1016/j.physa.2019.122272
H. Gunduz, Y. Yaslan and Z. Cataltepe, "Intraday prediction of Borsa Istanbul using convolutional neural networks and feature correlations," Knowledge-Based Systems, vol. 137, pp. 138–148, 2017. https://doi.org/10.1016/j.knosys.2017.09.023
E. Hoseinzade and S. Haratizadeh, "CNNpred: CNN-based stock market prediction using a diverse set of variables," Expert Systems With Applications, vol. 129, pp. 273–285, 2019. https://doi.org/10.1016/j.eswa.2019.03.029
S. Mishra, C. Bordin, K. Taharaguchi and I. Palu, "Comparison of deep learning models for multivariate prediction of time series wind power generation and temperature," Energy Reports, vol. 6, suppl. 3, pp. 273–286, 2019. https://doi.org/10.1016/j.egyr.2019.11.009
S. Totaro, A. Hussain and S. Scardapane, "A non-parametric softmax for improving neural attention in time-series forecasting," Neurocomputing, vol. 381, pp. 177–185, 2020. https://doi.org/10.1016/j.neucom.2019.10.084
X. Li, W. Shang and S. Wang, "Text-based crude oil price forecasting: A deep learning approach," International Journal of Forecasting, vol. 34, no. 4, pp. 1548–1560, 2019. https://doi.org/10.1016/j.ijforecast.2018.07.006
H. Abbasimehr, M. Shabani and M. Yousef, "An optimized model using LSTM network for demand forecasting," Computers & Industrial Engineering, vol. 143, no. 106435, pp. 1–13, 2020. https://doi.org/10.1016/j.cie.2020.106435
C.-W. Tsai, C.-H. Hsia, S.-J. Yang, S.-J. Liu and Z.-Y. Fang, "Optimizing hyperparameters of deep learning in predicting bus passengers based on simulated annealing," Applied Soft Computing Journal, vol. 88, no. 106068, pp. 1–9, 2020. https://doi.org/10.1016/j.asoc.2020.106068
H. Cui and J. Bai, "A new hyperparameters optimization method for convolutional neural networks," Pattern Recognition Letters, vol. 125, pp. 828–834, 2019. https://doi.org/10.1016/j.patrec.2019.02.009
S. Cheng, F. Lu, P. Peng and S. Wu, "Multi-task and multi-view learning based on particle swarm optimization for short-term traffic forecasting," Knowledge-Based Systems, vol. 180, pp. 116–132, 2019. https://doi.org/10.1016/j.knosys.2019.05.023
Y. E. Midilli and S. Elevli, "Optimization of Neural Networks with Response Surface Methodology: Prediction of Cigarette Pressure Drop," in 60th International Scientific Conference on Information Technology and Management Science of Riga Technical University, Riga, 2019. https://doi.org/10.1109/ITMS47855.2019.8940643
Suhartono, N. Suhermi and D. D. Prastyo, "Design of Experiment to Optimize the Architecture of Deep Learning for Nonlinear Time Series Forecasting," Procedia Computer Science, vol. 144, pp. 269–276, 2018. https://doi.org/10.1016/j.procs.2018.10.528
Y. E. Midilli and S. Parshutin, "Review for Optimisation of Neural Networks with Genetic Algorithms and Design of Experiments in Stock Market Prediction," Information Technology and Management Science, vol. 22, pp. 15–21, 2019. https://doi.org/10.7250/itms-2019-0003
L.-F. Hsieh, S.-C. Hsieh and P.-H. Tai, "Enhanced stock price variation prediction via DOE and BPNN-based optimization," Expert Systems with Applications, vol. 38, no. 11, pp. 14178–14184, 2011. https://doi.org/10.1016/j.eswa.2011.04.229
S. Sakarya, M. Yavuz, A. D. Karaoglan and N. Ozdemir, "Stock Market Index Prediction with Neural Network During Financial Crises: A Review on BIST-100," Financial Risk and Management Reviews, vol. 1, no. 2, pp. 53–67, 2015. https://doi.org/10.18488/journal.89/2015.1.2/89.2.53.67
A. Lasfer, H. El-Baz and I. Zualkernan, "Neural Network Design Parameters for Forecasting Financial Time Series," in 5th International Conference on Modeling, Simulation and Applied Optimization, IEEE, 2013. https://doi.org/10.1109/ICMSAO.2013.6552553
S. Alonso-Monsalve, A. L. Suárez-Cetrulo, A. Cervantes and D. Quintana, "Convolution on neural networks for high-frequency trend prediction of cryptocurrency exchange rates using technical indicators," Expert Systems With Applications, vol. 149, no. 113250, pp. 1–15, 2020. https://doi.org/10.1016/j.eswa.2020.113250
R. Dash, S. Samal, R. Dash and R. Rautray, "An integrated TOPSIS crow search based classifier ensemble: In application to stock index price movement prediction," Applied Soft Computing Journal, vol. 85, no. 105784, pp. 1–14, 2019. https://doi.org/10.1016/j.asoc.2019.105784
J. Cheng, L. Dong and M. Lapata, "Long Short-Term Memory-Networks for Machine Reading," Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, 2016. https://doi.org/10.18653/v1/D16-1053
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser and I. Polosukhin, "Attention Is All You Need," arXiv:1706.03762, 2017.
C.-L. Hwang, S.-J. Chen and F. Hwang, Fuzzy Multiple Attribute Decision Making: Methods. Berlin: Springer, 1992.
D. C. Montgomery, Statistical quality control: A modern introduction. New Jersey: John Wiley & Sons Singapore Pte. Ltd., 2013.
G. Derringer and R. Suich, "Simultaneous Optimization of Several Response Variables," Journal of Quality Technology, vol. 12, no. 4, pp. 214–219, 1980. https://doi.org/10.1080/00224065.1980.11980968
MetaQuotes, "Metatrader 5," MetaQuotes Software Corporation, 2000. [Online]. Available: https://www.metatrader5.com/en/trading-platform. [Accessed 8 5 2019].
DOI: 10.7250/itms-2020-0004
Refbacks
- There are currently no refbacks.
Copyright (c) 2020 Yunus Emre Midilli, Sergei Parshutin
This work is licensed under a Creative Commons Attribution 4.0 International License.