Research on Crop Disease Identification Methods Based on Multimodal Data Fusion and CLIP Model

Abstract

Crop disease is a major threat to food security, while current identification methods often suffer from low accuracy and weak generalization under complex field conditions. This paper proposes a multimodal crop disease identification model integrating image data and textual symptom descriptions. First, a multimodal dataset covering 12 common crop diseases is constructed, including 8,000 high-quality disease images and 8,000 symptom texts. Second, a Contrastive Language-Image Pre-training model is used to extract cross-modal features, and an attention mechanism is introduced to achieve deep fusion of image and text representations. Third, a multi-scale feature pyramid network is designed to capture disease characteristics at different granularity levels, while a focal loss function is introduced to address class imbalance. Experimental results show that the proposed method achieves 96.8% accuracy in the 12-category disease identification task, which is 8.3 percentage points higher than the single-modal method. The recall and F1 score reach 95.7% and 96.2%, respectively, and the inference speed reaches 42 frames per second. Ablation experiments show that the multimodal fusion module contributes a 4.5% performance improvement.