Published on February 20, 2014
HDF5 for NPOESS Data Products Alan M. Goldberg The MITRE Corporation email@example.com This work was performed under NOAA contract 50-SPNA-9-00010. Opinions expressed are those of the author, and do not represent the United States Government. Organization: W803 Project: 1400NT01-SE © 2002 The MITRE Corporation. All rights reserved.
The NPOESS Program ● Products ● Requirements ● HDF Implementation (Presentation by Chad Fox / Raytheon)
Data Flow Overview Field Terminals and Centrals NPOESS-Unique Collection NPOESS-Unique Processing Application Processing, Display, & Dissemination Demodulator Space Segment User Terminal Antenna RF Electronics C3S/DRR FT Processor Element IDPS Central Element External I/F Other Data Central
Data Delivery to Users ● Centrals * - National Environmental Satellite, Data, and Information Service Naval Oceanography Command Fleet Numerical Meteorology & Oceanography Command Air Force Weather Agency ● Field terminals * US government Private Worldwide users ● NESDIS Archives * ● Misc NASA (for NPP), SARSat, DCS, etc. -
Major Product Relationships Propagation Sensor Packetized Data Sensor Scene Environmental Phenomena Environmental Data Record 10011 01100 11100 Comm Raw Data Record Sensor Data Record Environment Model A0569F22 3B365CC 369B 707070FFF Sensor Model A0569F22 3B365CC 369B 707070FFF 10011 01100 11100
Product Types ● Major products - Raw data records * - Sensor Data Records * - Temperature Data Records * - Environmental data records * ● Other - Raw data - Auxiliary data - Ancillary Data - Calibration & correction data - Telemetry & housekeeping - Memory dumps and diagnostics
Delivery Requirements ● Operational *; platform independence; suitable for transmission, efficient conversion, and storage ● Standards based: Interoperability*, consistent with Joint Technical Architecture and National Spatial Data Infrastructure ● Rapid & available*: >95% delivered within 30 min. at Centrals; 15 min. goal ● Self-documenting ● Flexible for over 100 initial products ● Flexible for evolutionary or new sensors and algorithms ● User selectable aggregation (from one granule up to full orbit) ● Delivery: Push or pull; Assured; Secure* ● Efficient: 3 TB per day at each Central ● Consistent interface for delivery to Centrals*, Field terminals*, & long term archives* * key requirement
Schedule ● Interface with Centrals to be published (draft) in spring 2003. ● First deliverable version of IDPS to be ready in spring 2005 for NPP risk-reduction mission in spring 2006 ● Hardware specification for software support at field terminals to be published by Oct. 2005 ● Operational system to be ready in mid 2008 for first NPOESS launch in mid 2009.
Issues ● How much EOSDIS heritage to retain, if any, e.g. – whether to use EOS swath construct ● Develop an NPOESS profile to handle particular attributes of NPOESS data, e.g. variable length compressed packets in RDRs conical scan geometry ● Assure long-term stability of the standard ● Provide user support ● Suitability for archival use HDF5 is primarily defined by its API, not the format – – –
Data Processing Flow in IDPS Delivered Delivered Raw Data Raw Data Metadata Metadata Communications Model Metadata Metadata EDR Process Metadata Metadata Mission Data Mission Data Packets Packets RDR Process SDR Process Sensor Model Converted Converted Mission Mission Data Data Environ ment Model EnviEnvironment ronment Data Data Corr. Data Corr. Data Packets Packets SDRlike Process Auxiliary/ Auxiliary/ Telemtry/ Telemtry/ Hskpng Data Hskpng Data Packets Packets Ancillary Ancillary Data Data Ancillary Data Ancillary Data Packets Packets Format Format Format LRD Only RDR Converted Telemetry TDR/ SDR Local Product Interface to Users Format EDR
Raw Data Granule Contents ● CCSDS Application Packets, unprocessed, generally from one sensor over a short time interval ● Mission data as produced by the sensor or payload ● Telemetry ● Calibration data & Correction parameters ● Auxiliary data: spacecraft location & attitude
Sensor/Temperature Granule Contents ● Radiometrically corrected data, which is an estimate of the flux at the sensor aperture. ● Mission data by in-track position, cross-track position, channel, detector, etc. ● Georeferenced in spacecraft coordinates, & intersection with the ellipsoid ● Generally, not filtered, resampled, etc. ● TDRs are microwave SDRs without antenna pattern removed
Environmental Data Granule Contents ● SDRs processed with environmental models and ancillary data (by others) to produce estimates of environmental parameters ● Reported as cross-track / in-track values identified in earth coordinates unless resampling is specifically required -
Metadata - Scope ● Identifications: Name / Level / Type-subtype / Sequential number ● Description: Start-end date-time; Start-end orbit Subsatellite coordinates; Corner coordinates ● History: Space segment components / DRR components / Source files / Algorithm versions / Processing facility / Processing date-time ● Contents: Size / Granules Predecessor quality / Missing data / Out-of-bounds / Processing errors / Quality free text ● Formats: Data elements / Structures / Formats -
Metadata - Structure ● As goal, all metadata will be compliant with NSDI standards - <tag> = “parameter”, where <tag> is defined by FGDC or the NPOESS extension profile. ● Metadata will be hierarchical, as appropriate: - file-metadata ● granule_metadata – scan_metadata » point_metadata ● Metadata should map to the file naming convention ● Metadata will be included with associated data files, and accessible as a stand-alone file.
Data ● Hierarchical ● Sensor specific format ● “Internal” metadata, such as detailed geolocation, quality, annotation, illumination & view geometry
Why HDF5? ● Familiarity -- Environmental scientists already have experience with the standard, most recently from EOS products. ● Maturity -- HDF has shown its "staying power", and has been available long enough to have matured from user experiences. NASA, DOE, and others invested heavily in its development. ● Capability -- HDF was designed to manage large, compound data sets within high performance computing environments. HDF5 incorporates new features that are important for NPOESS. ● Compatibility -- HDF operates on multiple appropriate operating systems and languages: C, C++, Java and Fortran.
Why HDF5? (concluded) ● Availability -- HDF was developed in the public interest at NCSA, and is freely available. HDF also has many free, share and commercial supports tools available. ● Interoperability -- The DoD Joint Technical Architecture is in the process of accepting HDF as a standard for interoperability among DoD systems. ● Efficiency – Low overhead compared with legacy formats: GRIB and BUFR. Efficient indexing and subsetting. Support for data compression.
Notional Structure -- external metadata UniqueFileName.txt summary_ metadata file_metadata granule_ metadata granule_1 granule_n granule_2
Notional Structure -- HDF portion UniqueFileName.hdf root metadata data granule_1 granule_n granule_2 mission _data local_ metadata other
HDF5 for NPOESS Data Products May 12, 2015 Technology the-hdf-eos-tools-and-information-center
HDF5 for NPOESS Data Products Alan M. Goldberg The MITRE Corporation firstname.lastname@example.org Organization: W803 Project: 1400NT01-SE This work was performed ...
HDF5 NPOESS HDF5 NPOESS A “c” language demonstration exercise ... for each product in the "Data_Products" path of the input HDF5 file (H5Giterate)
... "HDF5 for NPOESS Data Products Alan M. Goldberg The MITRE Corporation Organization: W803 Project: 1400NT01-SE This work was performed."—
9 HDF5 for NPOESS Hierarchical Data Format 5 ... HDF5 for NPOESS Data Products Alan M. Goldberg The MITRE Corporation Organization: W803 Project: ...
HDF5 NPP Data Format ... HDF5 for NPP • Hierarchical Data Format 5 (HDF5) ... Data_Products
HDF Support for the ... encourage NPOESS products and applications to use HDF5 in ... talks on the NPOESS data representation in HDF5 using ...
HDF5 FOR NPOESS SENSOR AND ENVIRONMENTAL DATA RECORDS Richard Ullman 1, Ronald Andrews 2 1NASA Goddard Space Flight Center, Code 586, Greenbelt, MD ...