Optimization of Hyperparameters of BERT Model in Sentiment Analysis Using Genetic Algorithm
Subject Areas : Natural Language Processingshahla sadeghani 1 , mohammad jafar Tarokh 2 * , Mohammad Ali Afshar Kazemi 3
1 - Department of Information Technology Management, SR.C., Islamic Azad University, Tehran, Iran.
2 - Department of Industrial Engineering, K.N., Toosi Univesrsity of Technology, Tehran, Iran.
3 - Department of Industrial Management, CT.C., Islamic Azad University, Tehran, Iran.
Keywords: Sentiment analysis, BERT, genetic algorithm, hyperparameter optimization.,
Abstract :
Sentiment analysis, a vital branch of natural language processing (NLP), is progressing rapidly, and advanced models such as BERT are used to improve the accuracy of such analyses. However, tuning BERT’s hyperparameters remains a major challenge, directly influencing its performance. The methods used in previous research to optimize hyperparameters often face limitations in accuracy and efficiency. This study addresses an important research knowledge gap by using genetic algorithm (GA) to present a new approach to optimize the hyperparameters of BERT model in sentiment analysis with the aim of improving accuracy and efficiency. In this regard, this research is designed analytically and experimentally. Data were collected through Twitter API and analyzed after NLP. The BERT model’s hyperparameter optimization was performed using GA and the optimized models were evaluated based on criteria such as accuracy, precision, recall and F-1 score. The proposed algorithm was compared with other common methods such as vanilla BERT, Long Short-Term Memory (LSTM) and convolutional neural network (CNN). The results showed that the proposed model has significantly better performance than other methods, achieving 98.1% accuracy, 98.2% precision, 98.1% recall, and 98.15% F-1 score. In comparison, vanilla BERT model, LSTM and CNN scored 92.8%, 91.3% and 90.5% in accuracy, respectively. These results indicate that the use of GA in optimizing the hyperparameters of BERT model can effectively increase the accuracy and efficiency of sentiment analysis. This approach not only has applicability in various fields of text data analysis, but also can be used as an effective solution for future research in optimizing deep learning models.
[1] M. Li, L. Chen, J. Zhao, and Q. Li, “Sentiment analysis of Chinese stock reviews based on BERT model,” Applied Intelligence, vol. 51, no. 7, 2021, doi: 10.1007/s10489-020-02101-8.
[2] K. Naithani and Y. P. Raiwani, “Realization of natural language processing and machine learning approaches for text-based sentiment analysis,” Expert Syst, vol. 40, no. 5, 2023, doi: 10.1111/exsy.13114.
[3] A. Mewada and R. K. Dewang, “SA-ASBA: a hybrid model for aspect-based sentiment analysis using synthetic attention in pre-trained language BERT model with extreme gradient boosting,” Journal of Supercomputing, vol. 79, no. 5, 2023, doi: 10.1007/s11227-022-04881-x.
[4] X. Li, X. Wang, and H. Liu, “Research on fine-tuning strategy of sentiment analysis model based on BERT,” in 2021 IEEE 3rd International Conference on Communications, Information System and Computer Engineering, CISCE 2021, 2021. doi: 10.1109/CISCE52179.2021.9445882.
[5] R. Qasim, W. H. Bangyal, M. A. Alqarni, and A. Ali Almazroi, “A Fine-Tuned BERT-Based Transfer Learning Approach for Text Classification,” J Healthc Eng, vol. 2022, 2022, doi: 10.1155/2022/3498123.
[6] J. Martinez-Gil, “Optimizing readability using genetic algorithms,” Knowl Based Syst, vol. 284, 2024, doi: 10.1016/j.knosys.2023.111273.
[7] D. Choudhury and T. Acharjee, “A novel approach to fake news detection in social networks using genetic algorithm applying machine learning classifiers,” Multimed Tools Appl, vol. 82, no. 6, 2023, doi: 10.1007/s11042-022-12788-1.
[8] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Bidirectional encoder representations from transformers,” arXiv preprint arXiv:1810.04805, p. 15, 2018.
[9] H. Liu et al., “Use of BERT (Bidirectional Encoder Representations from Transformers)-Based Deep Learning Method for Extracting Evidences in Chinese Radiology Reports: Development of a Computer-Aided Liver Cancer Diagnosis Framework,” J Med Internet Res, vol. 23, no. 1, 2021, doi: 10.2196/19689.
[10] L. B. Cesar, M. Á. Manso-Callejo, and C. I. Cira, “BERT (Bidirectional Encoder Representations from Transformers) for Missing Data Imputation in Solar Irradiance Time Series †,” Engineering Proceedings, vol. 39, no. 1, 2023, doi: 10.3390/engproc2023039026.
[11] A. Areshey and H. Mathkour, “Transfer Learning for Sentiment Classification Using Bidirectional Encoder Representations from Transformers (BERT) Model,” Sensors, vol. 23, no. 11, 2023, doi: 10.3390/s23115232.
[12] L. Yang and A. Shami, “On hyperparameter optimization of machine learning algorithms: Theory and practice,” Neurocomputing, vol. 415, 2020, doi: 10.1016/j.neucom.2020.07.061.
[13] A. Pfob, S. C. Lu, and C. Sidey-Gibbons, “Machine learning in medicine: a practical introduction to techniques for data pre-processing, hyperparameter tuning, and model comparison,” BMC Med Res Methodol, vol. 22, no. 1, 2022, doi: 10.1186/s12874-022-01758-8.
[[14] S. Katoch, S. S. Chauhan, and V. Kumar, “A review on genetic algorithm: past, present, and future,” Multimed Tools Appl, vol. 80, no. 5, 2021, doi: 10.1007/s11042-020-10139-6.
[15] A. Lambora, K. Gupta, and K. Chopra, “Genetic Algorithm- A Literature Review,” in Proceedings of the International Conference on Machine Learning, Big Data, Cloud and Parallel Computing: Trends, Prespectives and Prospects, COMITCon 2019, 2019. doi: 10.1109/COMITCon.2019.8862255.
[16] M. Pota, M. Ventura, R. Catelli, and M. Esposito, “An effective bert-based pipeline for twitter sentiment analysis: A case study in Italian,” Sensors (Switzerland), vol. 21, no. 1, 2021, doi: 10.3390/s21010133.
[17] M. Pota, M. Ventura, H. Fujita, and M. Esposito, “Multilingual evaluation of pre-processing for BERT-based sentiment analysis of tweets,” Expert Syst Appl, vol. 181, 2021, doi: 10.1016/j.eswa.2021.115119.
[18] A. S. Talaat, “Sentiment analysis classification system using hybrid BERT models,” J Big Data, vol. 10, no. 1, 2023, doi: 10.1186/s40537-023-00781-w.
[19] U. Sirisha and B. S. Chandana, “Aspect based Sentiment and Emotion Analysis with ROBERTa, LSTM,” International Journal of Advanced Computer Science and Applications, vol. 13, no. 11, 2022, doi: 10.14569/IJACSA.2022.0131189.
[20] Y. Wu, Z. Jin, C. Shi, P. Liang, and T. Zhan, “Research on the Application of Deep Learning-based BERT Model in Sentiment Analysis,” arXiv preprint arXiv:2403.08217, 2024.
[21] A. Chiorrini, C. Diamantini, A. Mircoli, and D. Potena, “Emotion and sentiment analysis of tweets using BERT,” in CEUR Workshop Proceedings, 2021.
[22] S. Palani, P. Rajagopal, and S. Pancholi, “T-BERT--Model for Sentiment Analysis of Micro-blogs Integrating Topic Model and BERT,” arXiv preprint arXiv:2106.01097, 2021.
[23] H. Y. Lin and T. S. Moh, “Sentiment analysis on COVID tweets using COVID-Twitter-BERT with auxiliary sentence approach,” in Proceedings of the 2021 ACMSE Conference - ACMSE 2021: The Annual ACM Southeast Conference, 2021. doi: 10.1145/3409334.3452074.
[24] R. Anggrainingsih, G. M. Hassan, and A. Datta, “BERT based classification system for detecting rumours on Twitter,” arXiv preprint arXiv:2109.02975, 2021.
[25] A. Bello, S. C. Ng, and M. F. Leung, “A BERT Framework to Sentiment Analysis of Tweets,” Sensors, vol. 23, no. 1, 2023, doi: 10.3390/s23010506.
[26] H. A. Samir, L. Abd-Elmegid, and M. Marie, “Sentiment analysis model for Airline customers’ feedback using deep learning techniques,” 2023. doi: 10.1177/18479790231206019.
[27] S. Kardakis, I. Perikos, F. Grivokostopoulou, and I. Hatzilygeroudis, “Examining attention mechanisms in deep learning models for sentiment analysis,” Applied Sciences (Switzerland), vol. 11, no. 9, 2021, doi: 10.3390/app11093883.
[28] S. Seo, C. Kim, H. Kim, K. Mo, and P. Kang, “Comparative Study of Deep Learning-Based Sentiment Classification,” IEEE Access, vol. 8, 2020, doi: 10.1109/ACCESS.2019.2963426.