Automatic classification and segmentation of the sound track is an effective approach for video structuring. We use this method to parse soccer video. Firstly, based on the characteristics of soccer video, the paper summarizes three audio classes for soccer video, namely game-audio, advertisement-audio and studio-audio. Then it proposes a framework of audio classification and segmentation using Hidden Markov Model and combining some smoothing rules. We develop a 26-coefficients feature stream for HMM model. And the experimental studies indicate that the proposed framework is effective and robust.