In the fields of intelligent manufacturing, aerospace, and robotics, control systems often operate under unknown dynamics. This significantly limits the effectiveness of traditional model-based control methods. Reinforcement learning (RL), as a data-driven intelligent control approach, enables policy learning and optimization through interaction with the environment, showing great potential for solving optimal control problems in such model-unknown scenarios. This survey focuses on the issue of unknown dynamic models in continuous-time systems and first reviews the development of general reinforcement learning algorithms and their application in model-known scenarios through industrial examples and theoretical analysis methods. It also summarizes representative methods for model-unknown scenarios, such as model-based RL, off-policy integral RL, and Q-learning approaches. The survey also introduces Lyapunov-based theoretical analysis tools and important assumptions. It discusses cutting-edge topics such as RL under partial observability using large language models, safe RL, and stability and robustness enhanced RL, while highlighting the challenges faced by existing methods.