In order to reduce the false alarm rate and improve the detection efficiency for video smoke detection, the Y^dU^aV^a color model was proposed, which can characterize the spatial distribution and temporal variation of smoke. By using this color model to quickly screen the suspected smoke image blocks, the false alarm rate was reduced and the computing efficiency was improved. An improved MobileNetV3 network structure was proposed, which is aimed to extract deep features of images and to classify the suspected smoke image blocks so as to detect whether there is smoke in a video. The simulation results of video smoke detection show that this method has high accuracy, high detection frame rate, and low false alarm rate.