For solving the vibration control problem in complex excitation, the active control of magnetostrictive actuator of Jiles-atherton model in double-layer vibration isolation system was researched. Based on the traditional sliding mode control, a flexible neural network sliding mode control algorithm was proposed and the controller switching matrix was designed by the regularization method, then the updating formulas of the neural network weights and flexible mapping parameter were also established. Furthermore the control strategy was used for the active vibration control in double-layer vibration isolation system. Finally, the single frequency, multi frequency and random signal excitation were simulated and the results show that the flexible neural network sliding mode controller has a strong robustness and a good control effect.