Integrating the BERT-Deformer Architecture to Optimize Multimodal Translation of Japanese Literature
Main Article Content
Abstract
Accurate multimodal translation of Japanese literature requires effective integration of linguistic semantics and visual context while maintaining long-range dependency modeling and cultural fidelity. This study presents a BERT-Deformer architecture that combines a pre-trained Japanese BERT encoder with deformable attention to achieve hierarchical fusion of textual and visual information. The proposed framework employs deep contextual representations for literary texts and dynamically focuses on salient cross-modal features through adaptive reference-point sampling, enabling efficient processing of long sequences and culturally sensitive semantic alignment. Experimental results demonstrate superior performance over conventional Transformer-based approaches, achieving improvements in BLEU-4, METEOR, and BERTScore while exhibiting enhanced robustness for long-text translation and more accurate handling of culturally loaded expressions. Attention analysis further confirms that the model effectively exploits visual cues to strengthen semantic disambiguation and cross-modal consistency. Beyond literary translation, the proposed architecture provides a scalable multimodal information processing paradigm with potential relevance to semantic communication and intelligent electromagnetic information systems, where reliable cross-modal representation and context-aware information transmission are essential for high-fidelity knowledge delivery.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
References
P. Naveen and P. Trojovský, “Overview and challenges of machine translation for contextually appropriate translations,” Iscience, vol. 27, no. 10, pp. 110878, 2024, doi: 10.1016/j.isci.2024.110878.
L. Gu, “Translation of Japanese Literature Language and Natural Language Environment Understanding Based on Artificial Neural Network,” Journal of Environmental and Public Health, vol. 2022, no. 1, pp. 2015763, 2022, doi: 10.1155/2022/2015763.
P. S. Plyth and C. P. Craham, “Translation Affects Literary and Cultural Systems: How to Observe the Features of Translation? Applied Translation,” 2023; 14(1):29-37.
G. Corpas Pastor and L. Noriega-Santiáñez, “Human versus neural machine translation creativity: A study on manipulated MWEs in literature,” Information, vol. 15, no. 9, pp. 530, 2024, doi: 10.3390/info15090530.
S. M. Abdelhalim, A. A. Alsahil, and Z. A. Alsuhaibani, “Artificial intelligence tools and literary translation: a comparative investigation of ChatGPT and Google Translate from novice and advanced EFL student translators’ perspectives,” Cogent Arts & Humanities, vol. 12, no. 1, pp. 2508031, 2025, doi: 10.1080/23311983.2025.2508031.
C. Herold and H. Ney, “Improving long context document-level machine translation,” preprint arXiv, vol. 23, no. 06, pp. 05183, 2023, doi: 10.18653/v1/2023.codi-1.15.
L. Jin, J. He, J. May, and X. Ma, “Challenges in context-aware neural machine translation,” preprint arXiv, vol. 23, no. 05, pp. 13751, 2023, doi: 10.18653/v1/2023.emnlp-main.943.
T. Yao, S. Peng, L. Wang, Y. Li, and Y. Sun, “Cross-modality interaction reasoning for enhancing vision-language pre-training in image-text retrieval,” Applied Intelligence, vol. 54, no. 23, pp. 12230-12245, 2024, doi: 10.1007/s10489-024-05823-1.
L. Xiao, X. Wu, S. Yang, J. Xu, J. Zhou, and L. He, “Cross-modal fine-grained alignment and fusion network for multimodal aspect-based sentiment analysis,” Information Processing & Management, vol. 60, no. 6, pp. 103508, 2023, doi: 10.1016/j.ipm.2023.103508.
J. Wang, “Research on cultural translation based on neural network,” Mathematical Problems in Engineering, vol. 2022, no. 1, pp. 6330814, 2022, doi: 10.1155/2022/6330814.
S. Araghi and A. Palangkaraya, “The link between translation difficulty and the quality of machine translation: a literature review and empirical investigation,” Language Resources and Evaluation, vol. 58, no. 4, pp. 1093-1114, 2024, doi: 10.1007/s10579-024-09735-x.
I. D. Mienye, T. G. Swart, and G. Obaido, “Recurrent neural networks: A comprehensive review of architectures, variants, and applications,” Information, vol. 15, no. 9, pp. 517, 2024, doi: 10.3390/info15090517.
X. Xu, “Research on neural network machine translation model based on entity tagging improvement,” Mathematical Problems in Engineering, vol. 2022, no. 1, pp. 8407437, 2022, doi: 10.1155/2022/8407437.
A. Feng, I. Li, Y. Jiang, and R. Ying, “Diffuser: Efficient Transformers with Multi-hop Attention Diffusion for Long Sequences,” arXiv e-prints, vol. 22, no. 10, pp. 11794, 2022.
D. François, M. Saillot, J. Klein, T. F. Bissyandé, and A. Skupin, “Drop-in efficient self-attention approximation method,” Machine Learning, vol. 114, no. 6, pp. 139, 2025, doi: 10.1007/s10994-025-06768-3.
M. Pawłowski, A. Wróblewska, and S. Sysko-Roma´nczuk, “Effective techniques for multimodal data fusion: A comparative analysis,” Sensors, vol. 23, no. 5, pp. 2381, 2023, doi: 10.3390/s23052381.
F. Zhao, C. Zhang, and B. Geng, “Deep multimodal data fusion,” ACM computing surveys, vol. 56, no. 9, pp. 1-36, 2024, doi: 10.1145/3649447.
H. Yamauchi, T. Kajiwara, M. Katsurai, I. Ohmukai, and T. Ninomiya, “A Japanese Masked Language Model for Academic Domain,” Proceedings of the Third Workshop on Scholarly Document Processing, pp. 152-157, 2022.
E. Odle, Y. J. Hsueh, and P. C. Lin, “Semantic positioning model incorporating bert/roberta and fuzzy theory achieves more nuanced japanese adverb clustering,” Electronics, vol. 12, no. 19, pp. 4185, 2023, doi: 10.3390/electronics12194185.
A. R. Sajun, I. Zualkernan, and D. Sankalpa, “A historical survey of advances in transformer architectures,” Applied Sciences, vol. 14, no. 10, pp. 4316, 2024, doi: 10.3390/app14104316.
A. Rahali and M. A. Akhloufi, “End-to-end transformer-based models in textual-based NLP,” Ai, vol. 4, no. 1, pp. 54-110, 2023, doi: 10.3390/ai4010004.
K. Al-Hammuri, F. Gebali, A. Kanan, and I. T. Chelvan, “Vision transformer architecture and applications in digital health: a tutorial and survey,” Visual computing for industry, biomedicine, and art, vol. 6, no. 1, pp. 14, 2023, doi: 10.1186/s42492-023-00140-9.
Y. Wang, Y. Deng, Y. Zheng, P. Chattopadhyay, and L. Wang, “Vision transformers for image classification: A comparative survey,” Technologies, vol. 13, no. 1, pp. 32, 2025, doi: 10.3390/technologies13010032.
Y. Liu, D. Liu, and S. Zhu, “Bilingual-Visual Consistency for Multimodal Neural Machine Translation,” Mathematics, vol. 12, no. 15, pp. 2361, 2024, doi: 10.3390/math12152361.
E. Tian, Z. Zhu, F. Liu, Z. Li, R. Gu, and S. Zhao, “Multimodal Machine Translation Based on Enhanced Knowledge Distillation and Feature Fusion,” Electronics, vol. 13, no. 15, pp. 3084, 2024, doi: 10.3390/electronics13153084.
Y. Gan, Y. Fu, D. Wang, and Y. Li, “A novel approach to attention mechanism using kernel functions: Kerformer,” Frontiers in Neurorobotics, vol. 17, pp. 1214203, 2023, doi: 10.3389/fnbot.2023.1214203.
X. Song, Y. Tian, H. Liu, L. Wang, and J. Niu, “PPLA-Transformer: An Efficient Transformer for Defect Detection with Linear Attention Based on Pyramid Pooling,” Sensors, vol. 25, no. 3, pp. 828, 2025, doi: 10.3390/s25030828.
Y. Lu, J. Zeng, J. Zhang, S. Wu, and M. Li, “Learning confidence for transformer-based neural machine translation,” arXiv preprint arXiv, vol. 22, no. 03, pp. 11413, 2022, doi: 10.18653/v1/2022.acl-long.167.
T. Pearce, A. Brintrup, and J. Zhu, “Understanding softmax confidence and uncertainty,” arXiv preprint arXiv, vol. 21, no. 06, pp. 04972, 2021.
S. Lee, J. Lee, H. Moon, C. Park, J. Seo, S. Eo, et al., “A survey on evaluation metrics for machine translation,” Mathematics, vol. 11, no. 4, pp. 1006, 2023, doi: 10.3390/math11041006.
L. Yating, M. Afzaal, X. Shanshan, and D. A. S. El-Dakhs, “TQFLL: a novel unified analytics framework for translation quality framework for large language model and human translation of allusions in multilingual corpora,” Automatika, vol. 66, no. 1, pp. 91-102, 2025, doi: 10.1080/00051144.2024.2447652.
Y. Cui and M. Liang, “Automated scoring of translations with BERT models: Chinese and English language case study,” Applied Sciences, vol. 14, no. 5, pp. 1925, 2024, doi: 10.3390/app14051925.
Z. Xiao, X. Ning, and M. J. M. Duritan, “BERT-SVM: A hybrid BERT and SVM method for semantic similarity matching evaluation of paired short texts in English teaching,” Alexandria Engineering Journal, vol. 126, pp. 231-246, 2025, doi: 10.1016/j.aej.2025.04.061.
X. Zhang, W. Li, X. Wang, L. Wang, F. Zheng, L. Wang, et al., “A fusion encoder with multi-task guidance for cross-modal text-image retrieval in remote sensing,” Remote Sensing, vol. 15, no. 18, pp. 4637, 2023, doi: 10.3390/rs15184637.
U. R. Pol, P. S. Vadar, and T. T. Moharekar, “Hugging face: revolutionizing AI and NLP,” International Journal for Research in Applied Science and Engineering Technology, vol. 12, no. 8, pp. 1121-1124, 2024, doi: 10.22214/ijraset.2024.64023.