Purpose: A railway vehicle traveling on a stable rail line has a fixed amplitude and frequency, and hence it has constant sound values. However, line defects on the rail line alter this constant sound. Related to this, each defect type varies the reference sound value in a different way. In this study, audio signals originating from the three most common structural defects in rail systems were investigated with a deep neural network using audio to spectrogram conversion. Method: Short-term Fourier transform (STFT) was applied to sound signals collected from the rails and a convolutional neural network (CNN) model was developed to be able to acoustically detect the three most common types of defects (rail surface deformations, joint deformations, rail corrugations) in the railway superstructure at an early stage. The model learned five different classes, which are stable joint, stable rail surface, rail surface deformation, joint deformation, and rail corrugation with a train set (5000 observations). Results: The model obtained 89.37% accuracy with the validation data set (1000 observations). Finally, we evaluated the model with the test data (1000 observations) set that showed 87.3% accuracy. Conclusion: In rail transportation systems, high-frequency sound and vibration energy are formed by wheel–rail interaction. This study, which is easy to implement and shows the traceability of the changes occurring directly at the source of the noise, will provide a smart structural health monitoring way for rail systems in the future. For the next study, the data will be collected on the rail vehicle. A complete real-time system is a part of our ongoing research.