Abstract: The intelligent detection of rail defects is of great significance for rapidly determining the location and category of rail defect, improving detection efficiency, and enhancing operational safety. Addressing current issues in intelligent ultrasonic rail defect detection, such as missed detection, false alarms, and inaccurate results, this paper proposed a multi-modal heavy haul rail defect detection method based on multi-scale feature fusion. Building upon ultrasonic rail inspection data and B-scan image data, the method utilized time series processing methods such as sampling extension for the ultrasonic data and the MFF-SSD algorithm for the image data to detect target information. This enhanced defect localization and detection efficiency through richer feature dimensions. Furthermore, based on the characteristic representations and correlation levels of the different modal data, the final defect category was determined by integrating information from the multi-modal data. Experiments demonstrated that this method could reach a defect detection rate of 98.56%, with improved recognition speed.