Abstract: A computer program is worked out for modification design of helical gears by using Finite Element Method (FEM).At first the finite element mesh of gear tooth is auto generated without engagement of gears,therefore it can be divided more conveniently and uniformly.The flexibility matrixes of mesh nodes on working flanks of gears are calculated by FEM.Then,the contact lines and contact nodes on the plane of action are generated by computer according to the contact ratio of gear pair.The flexibility matrixes of contact nodes are obtained by interpolation between the mesh nodes on plane of action.Further,the deformation of shafts and bearings is added to the total flexibility matrix which is sum of the flexibility matrixes of both gears.Finally,considering minimization of transmitting error and uniformization of load density as objective functions,an optimal 3-D modification is solved by using mathematical programming technique.Moreover an experiment on modification of a helical gear pair has been done,which showed that the calculated results agreed with experimental data and the carrying behaviour of gear transmission was improved considerably.