The frequency difference of arrival (FDOA) estimation is the key point in a passive location system, and the estimation accuracy of FDOA determines the location performance. The FDOA estimation methods have been widely studied, but theoretical maximal accuracy of FDOA estimation is barely referred. To explore the ultimate performance of the FDOA estimation, the CRLB of frequency FDOA estimation accuracy was studied. On the basis of building a mathematical model, the quantitative relationship between the FDOA estimation accuracy and the parameters of received signal was investigated. By the means of mathematical deduction and numerical simulation, the algebraic expression of the CRLB was conducted in different application modes. Finally the characteristic of the CRLB was evaluated, and the conclusion can be used to direct the design of passive location system based on FDOA.