It is widely recognized that the shaft distance errors have vital effect on the accuracy and performance of anti-backlash gear transmission system. However，the essential mechanism and property is still undiscovered. Based on the contact constrained dynamics model of anti-backlash gear transmission system，the rigid-flexible coupling dynamics model of anti-backlash gear transmission system was established by adopting the ADAM/ Flex. Then，the rigid-flexible coupling dynamics simulation was carried out to study the vibration and frequency characteristics of the anti-backlash gear transmission system with different shaft distance errors. It is discovered that the spring torque，contact force and the friction torque will increase with the decrease of shaft distance，which will hamper the normal motion of the active gear and reduce the magnitude of rotation speed of the fixed shaft ; the main frequency of the main shaft will decrease with the increase of shaft distance，and the decrease will be sharper when the shaft distance is a little more larger than the standard centre distance. These discoveries will be helpful in the design and assembly of anti-backlash system.