The basic construction of H.324 is introduced first. Two DSP chips named TMS320VC5509 are selected to realize real time H.324 system by the analysis of calculating capacity and memory capacity needed in video and audio codec. In this design the resources in DSP are allocated reasonably, the key parts of video codec are optimized. In order to eliminate float operation in fixed point DSP,float-point DCT is replaced by integer transform. The experiment result is finally presented.