Abstract: There were several gear failures,in which the gear was broken into two,one of which is only a little smaller than one quarter of the gear.Several researchers had studied the cause of such gear failures and concluded that it was resonant vibrations.However the conclusion was not yet completely acceptable.In order to further ascertain the cause of such gear failures,the self-exciting vibration and instabilities were analyzed.The vibration instability of the disk-shaped plain gear was studied with the multiple scale's method of nonlinear perturbation method.The vibration response of a disk-shaped bevel gear was measured on a test rig.Both theoretical and experimental investigations show that there exist self-exciting vibrations on the gears with traveling wave vibration.Under certain conditions the forward traveling wave vibration of the driving gear and the backward traveling wave vibration of the driven gear are instable and cause the gear failures mentioned above.