Abstract: Based on the meshing condition of the gears,the constraint equation describing relations between the displacements in the centers and torsional angular displacements of the mating gears is derived.The coupled lateral and torsional vibration model of a rotor system with a gear coupling is developed with Lagrange's Equation and the constraint equation in rotating reference frames.The system of two Jeffcott rotors connected by a gear coupling has been analyzed numerically and the results show that the transient response frequency of aligned rotors approximately equals to the uncoupled model in lateral vibration,but the even-integer multiples of lateral vibration and the odd-integer multiples of torsional vibration are found out by the coupled model for the misaligned rotors.