Research on the modulation of the surface roughness to the RCS (radar cross section) has vital importance to the RCS scaled measurement for rough metal targets in the terahertz band. The changing laws of specular RCS of rough metal target with different surface roughness were studied. Based on the method of Kirchhoff approximation, combined with the theory of coherent and incoherent scattering, the method for predicting the specular RCS of rough metal target was proposed under the condition of small surface roughness. The effectiveness of the prediction method was validated by simulation experiments. The proposed method can achieve fast estimation of specular RCS for metal target with small surface roughness in the low terahertz band, and further predict the specular RCS for the scaled model with rough surface. This method also can provide effective RCS value for the comparison with the measurement result, which theoretically supports the study of RCS scaled measurements for rough targets in the terahertz spectrum.