It is of great necessity for the long term relative motion of formation flying to choose initial conditions and design the nature-periodic-relative rotating obits with the reference orbit eccentricity and the nonlinear relative motions. It is recognized that it yields considerable errors and is insufficient to employ Hill equations in choosing initial conditions and designing formation flying orbits without fuel consumption. With a consideration of the effects resulting from the reference orbit eccentricity and the nonlinearity relative motion, the numerical optimal technology was utilized in the present study. Results of numerical optimization can be used to determine the invariant-periodic-relative rotating conditions, thus gaining a good insight into formation flying. Numerical simulations demonstrate the correctness and effectiveness of the optimaized results.