To study the transverse combustion instability characteristics of the rocket combustor, numerical simulations of transverse combustion instability in a model rocket combustor were conducted based on the detailed chemical reaction mechanism (GRI Mech 3.0) and the flamelet-generated manifolds method. Accuracy of the numerical model was verified by comparing it with the experimental data. Pressure field was analyzed by the dynamic mode decomposition method, and the dynamic characteristics of the flow fields were investigated. Driving characteristics of combustion instability were quantitatively estimated by Rayleigh index. Result shows that the transverse combustion instability that occurred in the experiment can be effectively captured by the numerical model. Dominant frequency identified by the numerical study differ from the experimental value by less than 1%. Transverse pressure oscillations in the combustion chamber are coupled with that the longitudinal mode in the oxidizer post, leading to the pulsated propellant mass flow rate. Driving regions of combustion instability are mainly located on both sides of the combustion chamber, and the most marginal injectors played a critical role in keeping combustion instability. Heat release pulsations which periodically provide the energy source for the pressure oscillations are highly enhanced by the interactions between the propellant and the sidewall of the combustion chamber. And the combustion instability limit-cycle is formed.