advertisement

Using HDF5 Archive Information Package to preserve HDF-EOS2 data

100 %
0 %
advertisement
Information about Using HDF5 Archive Information Package to preserve HDF-EOS2 data
Technology

Published on February 17, 2014

Author: HDFEOS

Source: slideshare.net

Description

This one-year research project, funded by NOAA Climate Program Office (CPO) Scientific Data Stewardship (SDS), provides a solution to migrate data to a single standards-based archive format. Specifically, we investigate on how to store NASA ECS data and metadata into HDF5 Archival Information Packages (AIP). To achieve this, the HDF4 to HDF5 conversion tool has been enhanced so that converted ECS data can be read through the NetCDF4/CDM interface. In addition, metadata tools will be developed that convert ECS collection and granule level metadata to NOAA's collection level and NARA's METS standard. The enhanced HDF4 to HDF5 conversion tool has been released in May 2008 and it includes new functionality as the converted ECS data can be read through the NetCDF4 interface. We have tested 33 typical HDF-EOS2 swath, grid and point products at the National Snow and Ice Data Center (NSIDC). We also demonstrate the initial effort of the work to develop METS compliant metadata from granule metadata held in NASA's Earth Observing System (EOS) Data and Information System (EOSDIS) Core System (ECS).
advertisement

Storing EOS data using HDF5 Archival Information Package MuQun Yang (The HDF Group) Choonghwan Lee (The HDF Group) Ruth Duerr (NSIDC ) 10/16/2008 HDF and HDF-EOS Workshop XII 1

Definitions of some terms • CDM(Common Data Model) • Unification of the data models of OPeNDAP, netCDF and HDF5 • CF-Conventions • Convention of metadata providing spatial properties, among climate research community… • CLASS (Comprehensive Large Array-data Stewardship System) • Electronic library of NOAA environmental data 10/16/2008 HDF and HDF-EOS Workshop XII 2

Prerequisite • METS(Metadata Encoding & Transmission Standard) • Standard for encoding structural metadata • ISO-19115 • International Schema for describing geographic information • File-level Metadata • Metadata about the individual file or granule • Dataset-level Metadata • Metadata that applies to each and every granule/file in the whole data set(product) 10/16/2008 HDF and HDF-EOS Workshop XII 3

HDF5 Archive Information Package HDF5 Data file Metadata file Primary Schema METS Extension Schema |<mets> |---<dmdSec>----------------<MODS> |---<amdSec>--------------|--<techMD> | |--<rightsMD> | |--<sourceMD> |----<fileGrp> |----<structMap> HDF5 AIP Components p://www.hdfgroup.uiuc.edu/papers/papers/AIP/HDF5_AIP_White_Paper.pdf 10/16/2008 HDF and HDF-EOS Workshop XII 4

NOAA SDS Program CDM/NetCDF4 ISO-19115 NCDC:CLASS ECS to ISO19115 HDF5-AIP NetCDF4 / HDF5 Data METS NetCDF4/HDF5-data 10/16/2008 ECS to METS NSIDC/ECS Metadata H4to H5 NSIDC/ ECS HDF and HDF-EOS Workshop XII HDF4-data 5

Enhanced H4toH5 conversion tool • Convert HDF-EOS2 data to NetCDF4-compliant HDF5 data $ ./h4toh5 –eos –nc4 input.he2 output.nc4 $ ./h4toh5 –eos –nc4 input.he2 output.nc4 • Official release (2.0) can be found at http://hdfgroup.org/h4toh5/ 10/16/2008 HDF and HDF-EOS Workshop XII 6

Challenges to do the conversion • Retrieve geo-location information from HDFEOS2 data • Conform to NetCDF4 data model in the existing H4toH5 conversion tool • …… 10/16/2008 HDF and HDF-EOS Workshop XII 7

Challenges: Handle EOS - Grid • Grid lacks geolocation fields Data [4][12] [4][8] Lon[4][8] • Use predefined projections Lon[12] • • • • Geographic Geographic Sinusoidal Sinusoidal Polar stereographic … • New converter creates geolocation fields • HDF-EOS2 API GDij2ll() 10/16/2008 HDF and HDF-EOS Workshop XII 8

Challenges: Handle EOS - Swath • The size of geolocation fields can be different from data fields • New converter has to handle geolocation fields correctly 10/16/2008 HDF and HDF-EOS Workshop XII 9

Challenges in conforming to NetCDF4 New longitude has three columns Longitude has three columns Data field • Follow CF conventions field has two columns • Create two variables: NewLongitude and NewLatitude • Add to the data field an attribute coordinates=“NewLongitude NewLatitude” • Keep the original Latitude and Longitude 10/16/2008 HDF and HDF-EOS Workshop XII 10

Now some examples to show NetCDF4 files converted from EOS2 10/16/2008 HDF and HDF-EOS Workshop XII 11

A netCDF-4 file converted from EOS2 data at NSIDC 10/16/2008 HDF and HDF-EOS Workshop XII 12

A netCDF-4 file converted from EOS2 data at NSIDC 10/16/2008 HDF and HDF-EOS Workshop XII 13

Deliverables and future work • Deliverables 1. Enhanced HDF4 to HDF5 conversion tool http://hdfgroup.org/h4toh5/ 2. A validation tool to verify the correctness of the conversion Will be released soon! • Potential future work 1. Reduce the output file size 2. Improve the performance 10/16/2008 HDF and HDF-EOS Workshop XII 14

Acknowledgement This work was supported under NOAA Scientific Stewardship Program grant number NA07OAR4310286. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NOAA. 10/16/2008 HDF and HDF-EOS Workshop XII 15

Add a comment

Related presentations

Presentación que realice en el Evento Nacional de Gobierno Abierto, realizado los ...

In this presentation we will describe our experience developing with a highly dyna...

Presentation to the LITA Forum 7th November 2014 Albuquerque, NM

Un recorrido por los cambios que nos generará el wearabletech en el futuro

Um paralelo entre as novidades & mercado em Wearable Computing e Tecnologias Assis...

Microsoft finally joins the smartwatch and fitness tracker game by introducing the...

Related pages

The HDF Group Presentations - The HDF Group - Information ...

The HDF Group is a not-for-profit corporation ... - Introduction to HDF5 Data and ... - Using HDF5 Archive Information Package to preserve HDF ...
Read more

"Investigation of using HDF5 Archival Information Packages ...

1 11A.3 INVESTIGATION OF USING HDF5 ARCHIVAL INFORMATION PACKAGES (AIP) TO STORE NASA ECS DATA MuQun Yang*1, Ruth Duerr2, Choonghwan Lee 1 1The HDF Group,
Read more

Using HDF5 Archive Information Package to preserve HDF ...

Hier sollte eine Beschreibung angezeigt werden, diese Seite lässt dies jedoch nicht zu.
Read more

Hdf5 - Documents

Using HDF5 Archive Information Package to ... Sotring EOS data using HDF5 Archival Information Package ... Copyright © 2015 Powered By Documents.mx ...
Read more

HDFView User's Guide -- Introduction

... it simplifies the process of reading information from or writing data to ... Package <==> HDF-EOS2 ... information about HDF and HDF4 and HDF5 ...
Read more

Central Topic - HDF-EOS Tools and Information Center

Converts HDF-EOS2 files to HDF5 files. ... data in HDF5 files are susceptible to ... “Using HDF5 Archive Information Package to preserve HDF-EOS2 ...
Read more

ATMOSPHERIC SCIENCES AND CLIMATE APPLICATIONS USING HDF ...

P2.42 for more information about HDF5. In ... archives store and distribute data using HDF4 and ... Data Segment using HDF5. 4.
Read more

Agenda - HDF and HDF-EOS Workshop XII

HDF and HDF-EOS Workshop XII. October 15-17, ... Introduction to HDF5 Data and Programming ... Using HDF5 Archive Information Package to preserve HDF-EOS2 ...
Read more

HDF Group Projects - Information, Support, and Software

... and receive HDF5 data using an ... Group to archive even more satellite data from the ... HDF5 AIP HDF5 Archival Information Package ...
Read more