Aiming at the reliability-based design optimization problem of complex structural systems, an efficient optimization method based on subset simulation and Markov chain simulation in augmented space was proposed. Considering the reliability-based design optimization problem in which the design parameters were distributed parameters of basic random variables, the target failure probability was transformed into a posterior density function of the design parameters in the augmented space, obtained a set of initial failure samples in the whole design domain through subset simulation, and then adopted the efficient Markov chain simulation to generate more failure samples in the gradually smaller design domain under the sequential approximate optimization framework. The target posterior density function was estimated and updated, and the decoupling approach was used to solve the transformed optimization problem to finally obtain the optimum. Compared with the existing methods, the proposed method requires only one reliability analysis and can avoid local optimal solution, resulting in the global optimal solution. Examples were given to illustrate the applicability of the proposed method in engineering and its superiority in the accuracy and efficiency of analysis and calculation.