To meet the high memory bandwidth of SDR-oriented SIMD DSPs, a novel Vector Memory(VM) architecture is proposed. The VM consists of 16-way Vector Memory Blocks(VBs), and each VB contains two groups of multi-bank memory structure with low-order interleaved addressing. This structure aims at reducing the memory access conflicts, making best use of the bandwidth of the multi-bank memory, and realizing the parallel vector data access at the cost of low power consumption. Besides, a vector rearrangement unit is designed and implemented in the VM to support the 16-way unaligned SIMD vector access and share the VM space. Experimental results show that the proposed VM architecture can efficiently reduce or eliminate the data shuffling operations and speed up SDR applications.