Grid-to-grid routing algorithms for distributed hydrological modeling require large amount of computations which cannot be provided by sequential computation techniques. Parallel programming technology is necessary for large-scale and long-period simulations using grid-to-grid routing algorithms. There is currently little research on the parallelization of implicit finite difference based routing algorithms. A parallel implicit finite difference based grid-to-grid routing algorithm was presented, based on grids layering. In this algorithm, grids in the watershed were divided into different layers according to flow direction. The calculations of grids in a downstream layer cannot be performed until its upstream layers’ calculations were completed. The calculations of grids in the same layer are independent of each other, thus can be performed in parallel. So the parallelization strategy is to assign the calculation tasks of grids in the same layer to different CPU-cores to perform parallel computing. The algorithm was implemented by using the C++ programing language and the Open Multi-processing (OpenMP) Application Programming Interface (API) and was tested in the Qingshuihe watershed of Hebei Province under different amount of input data. The result shows that this parallel algorithm had good speedup and parallel efficiency. In addition, the case study showed that the parallel efficiencies were higher for simulations with large datasets than with small datasets and the up-down layering method had better performances than the down-up layering method.