Irregularities in the geometry and flexibility of railway crossings cause large impact forces, leading to rapid degradation of crossings. Precise stress and strain analysis is essential for understanding the behavior of dynamic frictional contact and the related failures at crossings. In this research, the wear and plastic deformation because of wheel–rail impact at railway crossings was investigated using the finite-element (FE) method. The simulated dynamic response was verified through comparisons with in situ axle box acceleration (ABA) measurements. Our focus was on the contact solution, taking account not only of the dynamic contact force but also the adhesion–slip regions, shear traction, and micro-slip. The contact solution was then used to calculate the plastic deformation and frictional work. The results suggest that the normal and tangential contact forces on the wing rail and crossing nose are out-of-sync during the impact, and that the maximum values of both the plastic deformation and frictional work at the crossing nose occur during two-point contact stage rather than, as widely believed, at the moment of maximum normal contact force. These findings could contribute to the analysis of non-proportional loading in the materials and lead to a deeper understanding of the damage mechanisms. The model provides a tool for both damage analysis and structure optimization of crossings.