Use BiLSTM with Attention Mechanism to Optimize the Accuracy of Word Meaning Correspondence in Technical Texts
Main Article Content
Abstract
In highly specialized and terminology-dense scientific and technical texts, existing word sense disambiguation (WSD) models struggle to adequately model the contextual semantic dependencies of polysemous words, especially in engineering domains where the same term may carry different technical meanings across contexts. To address this issue, this paper proposes a robust WSD model that integrates a bidirectional long shortterm memory network (BiLSTM ) with an attention mechanism, specifically designed for Chinese patent texts. First, a two-layer BiLSTM is used for bidirectional context modeling to capture long-range dependencies. Then, a multi-head attention mechanism uses dynamic weighting to highlight key semantic components, generating highly discriminative context vectors. Finally, a paraphrase alignment mechanism employs bilinear matching to align the context vectors with candidate paraphrase embeddings, thereby reducing semantic confusion. Experiments show that the model achieves a Top-1 accuracy of 88.6 % on high-frequency words, with an average paraphrase alignment similarity of 0.920. In perturbation tests, the average robustness index is 0.159, representing reductions of 62.7%, 53.9%, and 39.5% compared to Word2Vec+CNN, BiLSTM, and BERT, respectively. The method presented in this paper helps to enhance the accuracy and stability of word meaning recognition in technical texts, providing reliable support for knowledge mining and intelligent text processing in technical domains. Its terminology alignment is also useful for engineering corpora where antenna, wavepropagation and materials terms require context-sensitive interpretation.
Downloads
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
References
F. B. Abdullayeva, “Terminological Precision and Linguistic Challenges in Technical Translation,” American Journal of Philological Sciences, vol. 5, no. 4, pp. 333-337, 2025, doi: 10.37547/ajps/Volume05Issue04-82.
A. A. Khoroshilov, A. V. Kan, J. V. Nikitin, and A. D. A. Khoroshilov, “Machine phraseological translation of scientifictechnical texts based on the model of generalized syntagmas,” Automatic Documentation and Mathematical Linguistics, vol. 54, no. 2, pp. 79-91, 2020, doi: 10.3103/S0005105520020041.
O. O. Mezentseva and A. S. Kolomiiets, “Optimization of analysis and minimization of information losses in text mining,” Herald of Advanced Information Technology, vol. 3, no. 1, pp. 373-382, 2020, doi: 10.15276/hait.01.2020.4.
C. Makris, G. Pispirigos, and M. A. Simos, “Text semantic annotation: A distributed methodology based on community coherence,” Algorithms, vol. 13, no. 7, pp. 160-174, 2020, doi: 10.3390/a13070160.
D. Loureiro, K. Rezaee, M. T. Pilehvar, and J. Camacho-Collados, “Analysis and evaluation of language models for word sense disambiguation,” Computational Linguistics, vol. 47, no. 2, pp. 387-443, 2021, doi: 10.1162/coli_a_00405.
M. Q. Lu and Y. Y. Shen, “A text matching method based on Siamese network pre-trained language model,” Ensemble Technology, vol. 12, no. 2, pp. 53-63, 2023, doi: 10.12146/j.issn.2095-3135.20220817001.
H. Yu, Z. T. Liang, and H. Xie, “A review of online technology supply and demand text matching methods,” Information Science, vol. 39, no. 7, pp. 177-185, 2021, doi: 10.13833/j.issn.1007-7634.2021.07.024.
R. Kumar and S. C. Sharma, “Hybrid optimization and ontology-based semantic model for efficient text-based information retrieval,” The Journal of Supercomputing, vol. 79, no. 2, pp. 2251-2280, 2022, doi: 10.1007/s11227-022-04708-9.
P. Jiang and X. Cai, “A survey of text2-matching techniques,” Information, vol. 15, no. 6, pp. 332-384, 2024, doi: 10.3390/info15060332.
C. X. Zhang, S. Y. Pang, X. Y. Gao, J. Q. Lu, and B. Yu, “Attention neural network for bio-medical word sense disambiguation,” Discrete Dynamics in Nature and Society, vol. 2022, no. 1, pp. 1-14, 2022, doi: 10.1155/2022/6182058.
X. Y. Wang and H. S. Wang, “A review of text matching technology based on deep learning,” Information and Computers, vol. 32, no. 15, pp. 73-74, 2020, doi: 10.3969/j.issn.1003-9767.2020.15.027.
M. Hosseini, A. H. Rasekh, and A. Keshavarzi, “Improving clinical abbreviation sense disambiguation using attention-based Bi-LSTM and hybrid balancing techniques in imbalanced datasets,” Journal of Evaluation in Clinical Practice, vol. 30, no. 7, pp. 1327-1336, 2024, doi: 10.1111/jep.14041.
A. Salle and A. Villavicencio, “Understanding the effects of negative (and positive) pointwise mutual information on word vectors,” Journal of Experimental & Theoretical Artificial Intelligence, vol. 35, no. 8, pp. 1161-1199, 2023, doi: 10.1080/0952813x.2022.2072004.
F. A. O. Santos, T. D. Bispo, H. T. Macedo, and C. Zanchettin, “Morphological Skip-Gram: Replacing FastText characters n-gram with morphological knowledge,” Inteligencia Artificial, vol. 24, no. 67, pp. 1-17, 2021, doi: 10.4114/intartif.vol24iss67pp1-17.
Z. Yang, M. Ding, T. L. Huang, Y. K. Cen, J. S. Song, and B. Xu, “Does negative sampling matter? a review with insights into its theory and applications,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 8, pp. 5692-5711, 2024, doi: 10.1109/TPAMI.2024.3371473.
B. Liu, Y. Zhou, and W. Sun, “Character-level text classification via convolutional neural network and gated recurrent unit,” International Journal of Machine Learning and Cybernetics, vol. 11, no. 8, pp. 1939-1949, 2020, doi: 10.1007/s13042-020-01084-9.
R. Harizi, R. Walha, F. Drira, and M. Zaied, “Convolutional neural network with joint stepwise character/word modeling based system for scene text recognition,” Multimedia Tools and Applications, vol. 81, no. 3, pp. 3091-3106, 2022, doi: 10.1007/s11042-021-10663-z.
S. Minaee, N. Kalchbrenner, E. Cambria, N. Nikzad, M. Chenaghlu, and J. Gao, “Deep learning–based text classification: a comprehensive review,” ACM Computing Surveys (CSUR), vol. 54, no. 3, pp. 1-40, 2021, doi: 10.1145/3439726.
S. Johnson, S. Shen, and Y. Liu, “CWPC_BiAtt: Character–word–position combined BiLSTM-attention for Chinese named entity recognition,” Information, vol. 11, no. 1, pp. 45-63, 2020, doi: 10.3390/info11010045.
A. K. Nandanwar and J. Choudhary, “Semantic features with contextual knowledge-based web page categorization using the GloVe model and stacked BiLSTM,” Symmetry, vol. 13, no. 10, pp. 1772-1788, 2021, doi: 10.3390/sym13101772.
B. Jang, M. Kim, G. Harerimana, S. U. Kang, and J. W. Kim, “Bi-LSTM model to increase accuracy in text classification: Combining Word2vec CNN and attention mechanism,” Applied Sciences, vol. 10, no. 17, pp. 5841-5854, 2020, doi: 10.3390/app10175841.
F. J. Liu, N. Zhao, and G. Q. Zhu, “Cognitive difference text classification in online knowledge collaboration based on SA-BiLSTM hybrid model,” Scientific Reports, vol. 15, no. 1, pp. 22171-22184, 2025, doi: 10.1038/s41598-025-06914-w.
X. R. Yang, S. W. Zhao, R. X. Zhang, X. J. Yang, and Y. H. Tao, “BiLSTM_CNN text classification model combining selfattention and residual,” Journal of Computer Engineering & Applications, vol. 58, no. 3, pp. 172-180, 2022, doi: 10.3778/j.issn.1002-8331.2104-025.
M. J. Yuan, K. Z. Jiang, Y. Yang, and L. X. Hui, “MAC_BiLSTM text classification model combining self-attention and normalization,” Advances in Applied Mathematics, vol. 11, no. 10, pp. 7012-7025, 2022, doi: 10.12677/AAM.2022.1110744.
L. Shi, Y. Wang, Y. Cheng, and R. B. Wei, “A review of attention mechanism research in natural language processing,” Data Analysis and Knowledge Discovery, vol. 4, no. 5, pp. 1-14, 2020, doi: 10.11925/infotech.2096-3467.2019.1317.
T.-P. Shen, Q. Q. Meng, and Z. H. Zhan, “Named entity recognition of Chinese text based on attention mechanism,” Journal of Network Intelligence, vol. 8, no. 4, pp. 504-517, 2023, [Online]. Available: https://bit.nkust.edu.tw/~jni/2023/vol8/s2/13.JNI-0510.pdf.
J. Cheng, W. Tong, and W. Yan, “Capsule network improved multi-head attention for word sense disambiguation,” Applied Sciences, vol. 11, no. 6, pp. 2488-2501, 2021, doi: 10.3390/app11062488.
L. Enamoto, A. R. A. S. Santos, R. Maia, W. Li, and G. P. Rocha Filho, “Multi-label legal text classification with BiLSTM and attention,” International Journal of Computer Applications in Technology, vol. 68, no. 4, pp. 369-378, 2022, doi: 10.1504/ijcat.2022.125186.
Y. Y. Yan, F. A. Liu, X. Q. Zhuang, and J. Ju, “An R-transformer_BiLSTM model based on attention for multi-label text classification,” Neural Processing Letters, vol. 55, no. 2, pp. 1293-1316, 2023, doi: 10.1007/s11063-022-10938-y.
Y. F. Zheng, Z. H. Gao, J. Shen, and X. S. Zhai, “Optimizing automatic text classification approach in adaptive online collaborative discussion– A perspective of attention mechanism-based bi-lstm,” IEEE Transactions on Learning Technologies, vol. 16, no. 5, pp. 591-602, 2022, doi: 10.1109/TLT.2022.3192116.