Abstract: The rail will be damaged when the train is under heavy load operation. The traditional rail damage detection methods mainly rely on manual inspection or use of single modal characteristics for analysis, which has problems such as low accuracy, missing detection and reporting. Therefore, research is conducted on the detection method of rail damage in heavy-duty railways based on multimodal features. Firstly, images of rail damage in heavy-duty railways are collected, and the original images are preprocessed using histogram equalization. Then, the information in the image is transformed into modal vectors for feature extraction. The confidence matrix is selected to represent the distribution of different features in the image, and the feature modal elements in the rail damage image are decomposed. Finally, based on multimodal features, correlated damage feature moduli with correlation are annotated with loss feature labels to achieve the judgment and detection of heavy-duty rail damage. The experimental results showed that the proposed method can accurately located the damage location and had high detection accuracy for different levels of rail damage, with 5 000, 10 000,30 000 t heavy-duty railway rails as the test objects. It has practical application value.