Aiming at the difficulties in the choosing and planning in multi-period weapon systems development problems, an optimization simulation approach combining multi-objective optimization algorithm and reinforcement learning technique was proposed. A multi-objective optimization model was built to maximize the capability and minimize the cost of weapon portfolios in each period. Moreover, a solving algorithm based on the non-dominated sorting genetic algorithm-Ⅲ was presented to obtain the Pareto set in each period, based on which an optimization model for multi-period problem was built. The Q-Learning method, one of the reinforcement learning algorithms, searches within the Pareto set using two different ways for the selection of weapon portfolios in each period, whose outcome is used for the selection in the next period and the optimization of the portfolios over the entire planning horizon. An illustrative example was studied to demonstrate the effectiveness of the proposed model and hybrid algorithm, which can support the decision making on the weapons development and planning.