Practical Methods for Making HDF Data Usable by NetCDF-based Tools

50 %
50 %
Information about Practical Methods for Making HDF Data Usable by NetCDF-based Tools
Technology

Published on February 17, 2014

Author: HDFEOS

Source: slideshare.net

Description

A number of powerful NetCDF-based tools exist for visualizing and analyzing remote sensing data, such as IDV, McIDAS-V, Panoply, Ferret and GrADS. As a result, the GES DISC has been deploying a number of different strategies to make its native HDF, HDF-EOS2 and HDF-EOS5 data available in usable forms to these NetCDF tools. The most elaborate of these is an online, on-the-fly Web Service to convert HDF and HDF-EOS2 data into NetCDF-CF1. The end result is highly effective in making the data usable, even Level 2 swath data, which can be visualized in IDV. Unfortunately, the code is relatively convoluted and inflexible due to the difficulty of locating values in the HDF input file to describe the vertical dimension in CF1. (This could be made easier if HDF dimscales could be written through the HDF-EOS2 interface.) A similar approach has been to serve certain datasets through a Web Coverage Server developed by George Mason University, which is subject to many of the same issues. A third approach has been to use OPeNDAP to make data available to NetCDF-based tools. The HDF5 handler has proven to be effective for Level 3 gridded data in this respect at very low cost. These tools can access remote data directly through OPeNDAP, or the HDF5 data can be downloaded as a local NetCDF file. However, this is limited to HDF5 and does not work for Level 2 swath data. Recently, we have begun deploying a fourth approach, the Lats4D script running within GrADS. Though slower than our custom C++ converter, it supports (in principle) conversion of any Level 3 data that can be read in through GrADS, allowing for dimension definition through GrADS control files. Experimenting with Lats4D also led to the serendipitous discovery that NetCDF-based tools based on the netcdf-java library (e.g., IDV and Panoply) can actually read certain HDF files. If this capability could be generalized, the expense of netCDF conversion could be eliminated. Alas, the hierarchies written by HDF-EOS block this capability; we welcome ideas and contributions from the HDF and NetCDF communities to devising a solution to this impasse.

PRACTICAL METHODS FOR MAKING HDF DATA USABLE IN NETCDF-BASED TOOLS Chris Lynnes and Fan Fang Goddard Earth Sciences Data and Information Services Center HDF Workshop Thursday, November 5, 2009 5 Nov 2009 1

Panoply IDV GrADS Ferret HDF Workshop Thursday, November 5, 2009 5 Nov 2009 2 2

"And then a miracle occurs..." HDF Workshop Thursday, November 5, 2009 5 Nov 2009 Unidata IDV 3 3

IDV with HDF-EOS5 “self-describing format” HDF Workshop Thursday, November 5, 2009 5 Nov 2009 4 4

IDV with HDF-EOS5 “self-describing format” HDF Workshop Thursday, November 5, 2009 5 Nov 2009 4 4

Panoply w/HDF5 HDF Workshop Thursday, November 5, 2009 5 Nov 2009 5 5

Ferret and GrADS with HDF-EOS5 *Can be opened if user defines dimensions and uses xdfopen HDF Workshop Thursday, November 5, 2009 5 Nov 2009 6 6

Solution #1: Convert to netCDF • On-the-fly conversion to netCDF-3 • Custom C++ code • Acquiring the URL executes the conversion and downloads the file HDF Workshop Thursday, November 5, 2009 5 Nov 2009 7 7

Panoply w/converted AIRS L3 HDF Workshop Thursday, November 5, 2009 5 Nov 2009 8 8

IDV w/Converted AIRS L2 Surface Air Temperature HDF Workshop Thursday, November 5, 2009 5 Nov 2009 9 9

But... • Custom C++ code is expensive to maintain • Z dimension scale values are hard to find in AIRS data – – – – vdatas w/non-standard names external documents therefore: hard-coded in C++ class if only we could attach standard Z dimscales to the SDS • New class for each dataset HDF Workshop Thursday, November 5, 2009 5 Nov 2009 10 10

Solution #2: Lats4D Conversion • Essentially a GrADS script from Arlindo DaSilva, NASA/GSFC/ GMAO • Same on-the-fly mechanism as NetCDF converter • If you can read it into GrADS, you can convert it (usually) – Dimensions can be defined in GrADS control file – Limited to L3 and L4 gridded products – Handles both HDF-EOS and HDF • Also supports subsetting HDF Workshop Thursday, November 5, 2009 5 Nov 2009 11 11

TRMM Monthly (May 2009) in Panoply Lats4D Conversion + Subset HDF Workshop Thursday, November 5, 2009 5 Nov 2009 12 12

Solution #3: OGC Web Coverage Service • OGC WCS with netCDF/CF-1 profile – – – – Supports subsetting But: “prefers” interpolation to user’s resolution Requests are difficult for users to submit Works better for machine-to-machine interfaces • DataFed • DLR – Using GMU-developed WCS Server HDF Workshop Thursday, November 5, 2009 5 Nov 2009 13 13

AIRS L3 via WCS in Panoply wget -O airs_wcs.nc 'http://acdisc.sci.gsfc.nasa.gov/daac-bin/wcsAIRSL3?VERSION=1.0.0&SERVICE=WCS &REQUEST=getCoverage&FORMAT=NetCDF&COVERAGE=AIRX3STD:ascending:TotCldLiqH2O_A &BBOX=-179.5,-89.5,179.5,89.5&TIME=2007-04-07/2007-04-07/P1D&CRS=CRS:84&RESX=1.0&RESY=1.0' HDF Workshop Thursday, November 5, 2009 5 Nov 2009 14 14

Solution #4: OPeNDAP • HDF handlers – Released HDF5 version supports L3 grid – New HDF4 and HDF5 version supports • HDF4 • L2 swath HDF Workshop Thursday, November 5, 2009 5 Nov 2009 15 15

OPeNDAP Usability Results IDV Panoply Ferret GrADS ✓ Comments exc. AIRX3C2M ✓ latest version only AIRS L3 ✓ ✓ MODIS L3 ✓ ✓ OMI L3 ✓ ✓ ✓ ✓ TOMS L3 ✓ ✓ ✓ AIRS L2 ✓ ✓ ✓ ✓ x ✓ x x x ✓ x HIRDLS L2 MLS L2 MODIS L2 ✓ OMI L2 ✓ HDF Workshop Thursday, November 5, 2009 ✓ ✓ 5 Nov 2009 so far: MOD04, MOD07 exc. OMIBRO, OMHCO, OMOCLO, OMUVB 16 16

OMI L3 NearUV AAOD via OPeNDAP in Panoply HDF Workshop Thursday, November 5, 2009 5 Nov 2009 17 17

(but watch out for fast rendering in IDV) HDF Workshop Thursday, November 5, 2009 5 Nov 2009 18 18

IDV w/fast rendering off HDF Workshop Thursday, November 5, 2009 5 Nov 2009 19 19

Panoply w/MODIS L2 via new OPeNDAP HDF4 handler HDF Workshop Thursday, November 5, 2009 5 Nov 2009 20 20

Panoply w/AIRS L2 via new OPeNDAP HDF4 Handler • Cool..but who projected the data and how? HDF Workshop Thursday, November 5, 2009 5 Nov 2009 21 21

Solution #5: GrADS Data Server • DAP protocol, but data read by GrADS – Sort of like an OPeNDAP with a GrADS handler – If you can read it in in GrADS... • Includes time aggregation – (Not always a good thing) HDF Workshop Thursday, November 5, 2009 5 Nov 2009 22 22

MERRA via GDS in Panoply HDF Workshop Thursday, November 5, 2009 5 Nov 2009 23 23

Solution #6?: netcdf library MERRA HDF Subset from Lats4D in Panoply HDF Workshop Thursday, November 5, 2009 5 Nov 2009 24 24

netcdf libraries • netcdf-java library can read some HDF files – But not HDF-EOS... • Tried adding lat/long dimscales to an HDF-EOS2 grid, to no avail – Except for an MLS L3 HDF-EOS5 • (not a standard product) HDF Workshop Thursday, November 5, 2009 5 Nov 2009 25 25

Panoply, using netcdf-java w/ MLS L3, HDF-EOS5 HDF Workshop Thursday, November 5, 2009 5 Nov 2009 26 26

Can we make netcdf libraries read HDF-EOS? • Might require joint effort between UCAR, THG and tool developers to develop and debug... • OR, solution might emerge from HDF-EOS5 / netCDF-4 Augmentation effort • ...BUT – could make a lot of conversions unnecessary – would make data usage much easier for users HDF Workshop Thursday, November 5, 2009 5 Nov 2009 27 27

Trade Space OPeNDAP Cons Costly Data size expansion L3/L4 grid only Slow Interpolation Costly Reprojects L2 swath netcdf library L3/L4 only Slow HDF-EOS Custom C++ Conversion Lats4D Conversion WCS Pros L2 swath + L3/L4 grid Fast Easy to add HDF + HDF-EOS Subsetting L2 swath + L3/L4 grid Efficient Inexpensive GrADS Data Server HDF + HDF-EOS HDF Workshop Thursday, November 5, 2009 Enables many tools Obsoletes conversion 5 Nov 2009 28 28

Future of HDF-netCDF Tool Interoperability: Key Challenges • Level 1b data (calibrated radiance) – How should we fit the channel/wavelength dimension into CF1? • Level 2 in native swath coordinates • netCDF library direct access to HDF-EOS • Non-self-describing implementations of HDF – e.g., NPP HDF Workshop Thursday, November 5, 2009 5 Nov 2009 29 29

Add a comment

Related presentations

Related pages

Agenda - HDF and HDF-EOS Workshop XIII

HDF and HDF-EOS Workshop XIII. ... The HDF Group: Overview of HDF5 tools: Abstract: ... Practical Methods for Making HDF Data Usable by NetCDF-based Tools:
Read more

Making Data Usable: 2014 presentation at Datapalooza ...

Making Data Usable: ... Research methods to address gap See: ... Practical Methods for Making HDF Data Usable by NetCDF-based Tools.
Read more

Creating Usable Data - Technology - documents.mx

Creating Usable Data; Creating Usable Data Oct 30, 2014 Technology the-access-initiative. of 20 ...
Read more

Making Workflows More Usable By Biodiversity Researchers ...

System is processing data Please download to view Download 1 ... Download Making Workflows More Usable By Biodiversity Researchers. Transcript ...
Read more

1/20/2012 oxPlus Overview. Design Goals Use case for data ...

Slide 1 1/20/2012 oxPlus Overview Slide 2 Design Goals Use case for data federation Fun application usable by school Slide 3 oxServer XDI Messaging Graph ...
Read more

GO-ESSP Abstracts

Ingrid's data model is netcdf-based, ... making these responses re-usable. ... and governments require data and tools to navigate the maze of files and ...
Read more

Making Database Systems Usable H.V. Jagadish Adriane ...

System is processing data ... Download Making Database Systems Usable H.V. Jagadish Adriane Chapman Aaron Elkiss Magesh Jayapandian Yanyao Li Arnab Nandi ...
Read more

Software for Manipulating or Displaying NetCDF Data

Software for Manipulating or Displaying NetCDF Data. This document provides references to software packages that may be used for manipulating or ...
Read more