Abstract:The ASLR (address space layout randomization) is a defense mechanism to prevent the control-flow hijack. The lack of analysis of the impact of ASLR in existed automatic vulnerability analysis and exploit technologies makes the test cases difficult to be used in actual environment. Aimed at the defects of address randomization and features of its bypass technologies, an analysis method was proposed to deal with the vulnerability of ASLR based on program states transition. The FSM (finite states machine) was used to describe the transition of each key state on the program path, the constraints for some common scenes of memory leakage and control-flow hijack were built, and the vulnerability of ASLR was analyzed by solving the compatibility of memory leakage state constraints and control-flow hijack state constraints. Experimental results show that the proposed method can effectively detect ASLR bypass and control-flow hijack attacks caused by memory leakage, realize the automatic vulnerability analysis of ASLR, and improve the efficiency of software security analysis.