CyberGATE: Multiscale State-Space Fusion for Ocular and Vestibular Cybersickness Forecasting
Main Article Content
Abstract
Cybersickness is still a major usability and safety issue for virtual reality because discomfort may emerge gradually during exposure or appear suddenly when scene motion changes. Questionnaire-based evaluation is difficult to use as a continuous monitoring tool. This study presents CyberGATE, a multimodal time-series framework that estimates cybersickness from eye-tracking and head-motion signals collected by an HMD. The framework separates long-context representation and short-event modeling into two coordinated branches. The global branch uses multi-view embedding, Mamba sequence modeling, and KAN-based sparse experts to learn persistent temporal patterns, whereas the local branch applies decomposition-aware patch encoding and window attention to detect transient disturbances. Experiments on two public datasets show that CyberGATE reduces prediction error compared with recurrent, Transformer, and existing multimodal baselines. Component removal and sensitivity analyses further confirm that global dynamics, local responses, and adaptive routing all improve performance. These results indicate that ordinary HMD behavioral signals can support objective, real-time, and individualized cybersickness warning in practical VR applications.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
References
R. S. Kennedy, N. E. Lane, K. S. Berbaum, et al., “Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness,” The International Journal of Aviation Psychology, vol. 3, no. 3, pp. 203– 220, 1993.
D. Jeong and K. Han, “PRECYSE: Predicting Cybersickness using Transformer for Multimodal Time-Series Sensor Data,” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 8, no. 2, pp. 1–24, 2024.
B. Keshavarz, K. Peck, S. Rezaei, et al., “Detecting and predicting visually induced motion sickness with physiological measures in combination with machine learning techniques,” International Journal of Psychophysiology, vol. 176, pp. 14–26, 2022.
N. Tian and R. Boulic, “Who says you are so sick? An investigation on individual susceptibility to cybersickness triggers using EEG, EGG and ECG,” IEEE Transactions on Visualization and Computer Graphics, vol. 30, no. 5, pp. 2379–2389, 2024.
B. Fu, W. Chu, C. Gu, et al., “Cross-modal guiding neural network for multimodal emotion recognition from EEG and eye movement signals,” IEEE Journal of Biomedical and Health Informatics, vol. 28, no. 10, pp. 5865–5876, 2024.
J. Sameri, H. Coenegracht, S. Van Damme, et al., “Physiology-driven cybersickness detection in virtual reality: a machine learning and explainable AI approach,” Virtual Reality, vol. 28, no. 4, Art. no. 174, 2024.
E. Chang, H. T. Kim, and B. Yoo, “Predicting cybersickness based on user’s gaze behaviors in HMD-based virtual reality,” Journal of Computational Design and Engineering, vol. 8, no. 2, pp. 728–739, 2021.
Y. Nam, U. Hong, H. Chung, et al., “Eye movement patterns reflecting cybersickness: evidence from different experience modes of a virtual reality game,” Cyberpsychology, Behavior, and Social Networking, vol. 25, no. 2, pp. 135–139, 2022.
A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” arXiv preprint arXiv:2312.00752, 2024.
Z. Liu, Y. Wang, S. Vaidya, et al., “KAN: Kolmogorov-Arnold Networks,” arXiv preprint arXiv:2404.19756, 2024.
X. Xu, C. Chen, Y. Liang, et al., “SST: Multi-scale hybrid Mamba-Transformer experts for long-short range time series forecasting,” arXiv preprint arXiv:2404.14757, 2024.
H. Oh, W. Son, J. Park, et al., “Cybersickness and its severity arising from virtual reality content: a comprehensive study,” Sensors, vol. 22, no. 4, Art. no. 1314, 2022.
J. T. Reason and J. J. Brand, Motion Sickness. London: Academic Press, 1975.
G. E. Riccio and T. A. Stoffregen, “An ecological theory of motion sickness and postural instability,” Ecological Psychology, vol. 3, no. 3, pp. 195–240, 1991.
P. Kourtesis, A. Papadopoulou, and P. Roussos, “Cybersickness in virtual reality: the role of individual differences, effects on cognitive functions, motor skills, and intensity differences during and after immersion,” Virtual Worlds, vol. 3, no. 1, Art. no. 4, 2024.
R. Hussain, M. Chessa, and F. Solari, “Mitigating cybersickness in virtual reality systems through foveated depth-of-field blur,” Sensors, vol. 21, no. 12, Art. no. 4006, 2021.
I. Lukacova, B. Keshavarz, and J. F. Golding, “Measuring the susceptibility to visually induced motion sickness and its relationship with vertigo, dizziness, migraine, syncope and personality traits,” Experimental Brain Research, vol. 241, pp. 1381–1391, 2023.
Z. T. Gu, D. Ding, Y. T. Chen, et al., “Evaluation and mitigation methods for virtual reality motion sickness,” Journal of Zhejiang University (Science Edition), vol. 52, no. 1, pp. 37–46, 2025.
E. Krokos and A. Varshney, “Quantifying VR cybersickness using EEG,” Virtual Reality, vol. 26, no. 1, pp. 77–89, 2022.
G. Zerveas, S. Jayaraman, D. Patel, et al., “A Transformer-based framework for multivariate time series representation learning,” in Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2021, pp. 2114–2124.
M. Pang, H. Wang, J. Huang, et al., “Multi-scale masked autoencoders for cross-session emotion recognition,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 32, pp. 1637–1646, 2024.
R. K. Kundu, R. Islam, J. Quarles, et al., “LiteVR: interpretable and lightweight cybersickness detection using explainable AI,” arXiv preprint arXiv:2302.03037, 2023.
Y. Zhu, T. Li, and Y. Wang, “Real-time cross-modal cybersickness prediction in virtual reality,” arXiv preprint arXiv:2501.01212, 2025.
Y. Choi, D. Jeong, B. Kim, et al., “Early prediction of cybersickness in virtual reality using multimodal sensor data,” in Proceedings of the ACM Symposium on Applied Perception, 2024, pp. 1–10.
Y. Long, X. Zhang, H. Li, et al., “Toward accurate cybersickness prediction in virtual reality,” Sensors, vol. 25, no. 18, Art. no. 5606, 2025.
A. N. Ramaseri-Chandra, J. M. Bogle, and S. M. Peterson, “Exploring the feasibility of head-tracking data for cybersickness prediction in virtual reality,” Electronics, vol. 14, no. 3, Art. no. 502, 2025.
S. S. Yeo, S. Y. Park, and S. H. Yun, “Investigating cortical activity during cybersickness by fNIRS,” Scientific Reports, vol. 14, no. 1, Art. no. 8093, 2024.
R. Islam, S. Ang, and J. Quarles, “Towards forecasting the onset of cybersickness by fusing physiological, head-tracking and eye-tracking with multimodal deep fusion network,” in Proceedings of the IEEE International Symposium on Mixed and Augmented Reality, 2022, pp. 121– 130.
L. Chiparova and V. Popov, “Kolmogorov-Arnold Networks for system identification of first- and second-order dynamic systems,” Engineering Proceedings, vol. 100, no. 1, Art. no. 100059, 2025.
Q. Yu, Y. Xia, Y. Bai, et al., “Glance-and-gaze vision transformer,” arXiv preprint arXiv:2106.02277, 2021.
J. Zhang, Y. Wang, X. Liu, et al., “How different text display patterns affect cybersickness in augmented reality environments,” Scientific Reports, vol. 14, Art. no. 12679, 2024.
B. U. Demirel, A. H. Dogan, J. Rossie, et al., “Beyond subjectivity: continuous cybersickness detection using EEG-based multitaper spectrum estimation,” IEEE Transactions on Visualization and Computer Graphics, vol. 31, no. 5, pp. 2525–2534, 2025.
Y. Wang, J. R. Chardonnet, and F. Merienne, “VR sickness prediction for navigation in immersive virtual environments using a deep long short-term memory model,” in Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces, 2019, pp. 1874–1881.
H. Zhou, S. Zhang, J. Peng, et al., “Informer: beyond efficient Transformer for long sequence time-series forecasting,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 12, pp. 11106–11115, 2021.
M. Yalcin, A. Halbig, M. Fischbach, et al., “Automatic cybersickness detection by deep learning of augmented physiological data from off-the-shelf consumer-grade sensors,” Frontiers in Virtual Reality, vol. 5, Art. no. 1364207, 2024.
R. Islam, K. Desai, and J. Quarles, “Cybersickness prediction from integrated HMD’s sensors: a multimodal deep fusion approach using eye-tracking and head-tracking data,” in Proceedings of the IEEE International Symposium on Mixed and Augmented Reality, 2021, pp. 31–40.
D. Jeong, S. Paik, Y. Noh, et al., “MAC: multimodal, attention-based cybersickness prediction modeling in virtual reality,” Virtual Reality, vol. 27, no. 3, pp. 2315–2330, 2023.
M. Yalcin, A. Halbig, M. Fischbach, et al., “Automatic cybersickness detection by deep learning of augmented physiological data from off-the-shelf consumer-grade sensors,” Frontiers in Virtual Reality, vol. 5, Art. no. 1364207, 2024.
R. Li, Y. Wang, H. Yin, et al., “A deep cybersickness predictor through kinematic data with encoded physiological representation,” in Proceedings of the IEEE International Symposium on Mixed and Augmented Reality, 2023, pp. 1132–1141.