| 引用本文: | 张帅,李满春,陈振杰,等.栅格地理数据并行格式转换引擎.[J].国防科技大学学报,2015,37(5):9-14.[点击复制] |
| ZHANG Shuai,LI Manchun,CHEN Zhenjie,et al.Parallel geo-raster data conversion engine[J].Journal of National University of Defense Technology,2015,37(5):9-14[点击复制] |
|
| |
|
|
| 本文已被：浏览 8511次 下载 6219次 |
| 栅格地理数据并行格式转换引擎 |
| 张帅1,2,3,4, 李满春1,2, 陈振杰1,2, 黄涛1,2, 姜晓慧5 |
|
(1. 南京大学 地理与海洋科学学院， 江苏 南京 210023;2.
2. 南京大学 地理信息技术江苏省重点实验室， 江苏 南京 210023;3.
3. 伊利诺伊大学香槟分校 地理系, 伊利诺伊 61801;4.
4. 伊利诺伊大学香槟分校 国家超级计算机应用中心, 伊利诺伊 61801;5. 国电南瑞科技股份有限公司， 江苏 南京 211106)
|
| 摘要: |
| 如今大规模地理数据正在社会各个部门和组织中迅速积累，但是由于部门利益和历史沿袭等原因，大规模地理数据共享仍然极具挑战，相应共享技术需求仍然极其旺盛。作为地理数据共享的基础方式之一，传统单机地理数据格式转换技术，一方面受限于磁盘读写及带宽瓶颈，另一方面面对日趋庞大的数据规模，已很难满足实际应用需求。因此提出一种针对栅格地理数据的并行格式转换引擎，采用高性能计算集群环境支持大规模栅格地理数据转换共享，大幅降低了大规模栅格地理数据转换过程的时间成本。栅格地理数据并行格式转换引擎采用基于公共接口的设计理念，框架灵活、具有良好的扩展性，支持地理数据格式的读写自定义以及新数据格式添加，能够实现接入数据格式间的任意两两转换。为验证引擎框架及其处理效率，在Lustre并行集群环境下以格网数据交换格式（国家地理空间数据交换格式）向常见栅格地理格式的转换为示例进行了测试实验。结果表明，栅格地理数据并行格式转换引擎能够在8个节点Lustre集群中达到7.54的良好并行加速比。 |
| 关键词: 栅格数据格式转换 并行计算 高性能计算 地理计算 Lustre并行文件系统 |
| DOI：10.11887/j.cn.201505002 |
| 投稿日期：2015-03-02 |
| 基金项目:国家863计划资助项目 (2011AA120301)；南京大学研究生科研创新基金资助项目(2013CL09) |
|
| Parallel geo-raster data conversion engine |
| ZHANG Shuai1,2,3,4, LI Manchun1,2, CHEN Zhenjie1,2, HUANG Tao1,2, JIANG Xiaohui5 |
|
(1. School of Geographic and Oceanographic Sciences, Nanjing University, Nanjing 210023, China;2.
2. Jiangsu Key Lab of Geographical Information Technology, Nanjing University, Nanjing 210023, China;3.
3. Department of Geography, University of Illinois at UrbanaChampaign, Illinois 61801, United States;4.
4. National Center for Supercomputing Applications, University of Illinois at UrbanaChampaign, Illinois 61801, United States;5. Nari Technology Development Limited Company, Nanjing 211106, China)
|
| Abstract: |
| Large scale geo-raster data have been accumulated all over the world in different departments and organizations during the past decades, but quite often in a variety of data formats, resulting in geospatial data sharing as an everlasting headache. Despite of various methodologies created, geospatial data conversion has always been a fundamental and efficient way for geospatial data sharing. However, as the size of data tends to be larger and larger, the methodology which was bounded by limited disk data transfer rate and bandwidth, needs a re-write and up- grade. A parallel geo-raster data conversion engine (PGRCE) was proposed to deal with massive geo-raster data sharing efficiently by utilizing high performance computing technologies. PGRCE was designed in an extendable and flexible framework, and was capable of customizing the way of reading and writing of particular spatial data formats. An experiment, in which geo raster data in the CNSDTF-DEM format (Raster spatial data defined in Chinese Geospatial Data Transfer Format Standard) were transferred using PGRCE in a parallel file system (Lustre), were conducted to validate the engine framework and its performance. Results show that PGRCE can achieve a 7.54 speedup on a Luster cluster of 8 nodes. |
| Keywords: geo-raster data format conversion parallel computing high performance computing geocomputation Lustre cluster |
|
|
|
|
|
