In order to access independent memory space of CPU and GPU directly from opposite directions, an effective approach to accessing unified memory address space of heterogeneous kilo-cores system is proposed, which is implemented by building a unified 3-level Cache and tagging blocks in Cache, and optimizing the algorithms of modifying the states of blocks. Therefore, the heterogeneous kilo-cores system avoids significant overhead of accessing memory instead of that in current discrete hybrid computer system equipped with GPUs by PCI-E. According to the results of experiments from partial programs of Rodinia benchmarks, a maximal speedup by 9.8x and maximal decrease of load/store instructions by 90% are gained. In conclusion, it’s certified that our solution is effective to decrease overhead of transferring data among computing units in heterogeneous system and significantly enhance the whole system computing performance. 