Abstract: A gear transmission system exhibits a coupling nonlinear vibration due to the nonlinearity of gear meshing,bearing and gearbox supporting system stiffnesses incorporated with the modulating of gear and bearing as well as the nonlinear coupling of rotary wings.The dynamic signals of the system appear of complicated non-Gauss and nonlinear characteristics.A new method of modern signal processing-the higher order cumulant was adopted.The nonlinear modulating characteristics of the system were monitored by determining the dynamic response of the transmission system and helicopter rotary wings.The dynamic natural frequencies of the transmission system and the rotary wings were obtained by analyzing in the cumulant domain.Its comparison with the power spectrum method and the time-series analysis (ARMA) identification Method shows that the higher order cumulant method is supperior in noise inhibiting,higher identification accuracy and nonlinear characteristic extracting of the system.