Published on February 18, 2014
NPP/ NPOESS Product Data Format Richard E. Ullman NOAA/NESDIS/IPO • NASA/GSFC/NPP Algorithm Division • System Engineering Data/Information Architecture richard. firstname.lastname@example.org NPOESS•INTEGRATED PROGRAM OFFICE
Scope • • The file profile described in this presentation applies to all NPOESS delivered environmental remote sensing products of the following types: – – – – – – SDR • Sensor Data Record TDR • Temperature Data Record EDR • Environmental Data Record IP • Intermediate Product ARP • Application Related Product GEO • Geolocation This profile does not apply to other types such as: – RDR • Raw Data Record – DDR • Data Delivery Record – Mission notices, status notices, documentation, software deliveries, etc NPOESS•INTEGRATED PROGRAM OFFICE 2 The HDF Workshop 6, 2007 Nov
HDF5 for NPOESS • • • • • • Hierarchical Data Format 5 (HDF5) is the format for delivery of processed products from the National Polar-orbiting Operational Environmental Satellite System (NPOESS) and for the NPOESS Preparatory Program (NPP). HDF5 is a general purpose library and file format for storing scientific data. Two primary objects: • Dataset, a multidimensional array of data elements • Group, a structure for organizing objects Efficient storage and I/O, including parallel I/O. Free, open source software, multiple platforms. Data stored in HDF5 is used in many fields from computational fluid dynamics to film making. Data can be stored in HDF5 in an endless variety of ways, so it is important to standardize how NPOESS product data is organized in HDF5. NPOESS•INTEGRATED PROGRAM OFFICE 3 The HDF Workshop 6, 2007 Nov
Format Strengths • Straight HDF5. • Consistent HDF5 group structure • Allows for flexible temporal aggregation – No need for additional libraries. – Organization for each product is the same as all others. – Data “payload” is always in a product group within All_Data group. – Granules are appended by extending dataset dimension. NPOESS•INTEGRATED PROGRAM OFFICE 4 The HDF Workshop 6, 2007 Nov
Format Challenges • • • • • Geolocation appears in a separate product group and may be in separate HDF5 file. Field metadata, used to interpret data (similar to netCDF CF) are in separate product profile file. Quality flags must be parsed before they can be interpreted. Information needed for un-scaling scaled integers is not obvious. HDF5 indirect reference link API, used to link metadata to the data in NPOESS’ use is complex and not supported by all analysis COTS implementations. NPOESS•INTEGRATED PROGRAM OFFICE 5 The HDF Workshop 6, 2007 Nov
Information Model UML Diagram NPOESS•INTEGRATED PROGRAM OFFICE 6 The HDF Workshop 6, 2007 Nov
An Example Product Group • In this example product group: – – – – – – – Five datasets constitute the product. There are two common dimensions. There are three congruent datasets. Two datasets contain scale and offset values. One dataset contains quality flags by element. There are two granules in this aggregation. Dimension “alongTrack” crosses the “granule boundary.” NPOESS•INTEGRATED PROGRAM OFFICE 7 The HDF Workshop 6, 2007 Nov
Example extracted from VIIRS Sea Surface Temperature EDR Field Name Description Data Type Dimensions Units BulkTemp Sea Surface Bulk Temperature 16-bit unsigned integer [ N*768, 3200 ] Kelvin / Unitless SkinTemp Sea Surface Skin Temperature 16-bit unsigned integer [ N*768, 3200 ] Kelvin / Unitless QF1_VIIRSSSTEDR Land/Water Background 1-bit [N*768, 3200 ] Unitless SST Skin Quality 2-bit Unitless SST Bulk Quality 2-bit Unitless Aerosol Correction 3-bit Unitless Bulk SST Scale 32-bit float Bulk SST Offset 32-bit float Skin SST Scale 32-bit float Skin SST Offset 32-bit float SSTBulkFactors SSTSkinFactors [ N*2] Unitless Kelvin [ N*2] Unitless Kelvin N in Dimension is number of granules NPOESS•INTEGRATED PROGRAM OFFICE 8 The HDF Workshop 6, 2007 Nov
Example Product Group NPOESS Product Group alo Tr ng ac k crossTrack Granule 0 BulkTemp Granule 0 SSTBulkFactors Granule 1 SkinTemp Granule 1 S O S O Granule 0 QF1_VIIRSSSTEDR Granule 1 NPOESS•INTEGRATED PROGRAM OFFICE Granule 0 Granule 1 SSTSkinFactors S O S O Granule 0 Granule 1 9 The HDF Workshop 6, 2007 Nov
Dimensions • • • • • Dimensions are defined for each field. Fields are related by congruency and common dimensions. Common dimensions are given the same name. One dimension crosses the granule boundary. When multiple granules are “aggregated” the “granule boundary” dimension is extended. Dimension names and attributes are provided in the product profile. NPOESS•INTEGRATED PROGRAM OFFICE 10 The HDF Workshop 6, 2007 Nov
Scaled Integer Storage • • • • • For storage efficiency floating point data values may be stored as scaled integers. To re-generate the data value, the dataset element must be multiplied by a supplied scale factor and an integer offset added. The scale factor and offset are provided, one pair for each granule as a separate dataset. The scale and offset value is the same for all granules produced with a given version of an algorithm - not dynamic scaling. The fact that a dataset is a scaled value and the association between the data dataset and the scale factor dataset is contained in the product profile. NPOESS•INTEGRATED PROGRAM OFFICE 11 The HDF Workshop 6, 2007 Nov
Quality Flags by Element • • • Most NPOESS products contain multiple indicators of quality on an element by element basis. Quality flags are associated by congruency (shared dimension) with a data array. Multiple Flags of less than 8-bits are “packed” into structures aligned on 8-bit boundaries. 1 1 Quality Flags 0 1 0 1-bit 2-bit 2-bit 0 0 3-bit 1 NPOESS•INTEGRATED PROGRAM OFFICE 12 The HDF Workshop 6, 2007 Nov
Geolocation • • • Geolocation products are constructed using the same conventions as SDRs and EDRs. Geolocation datasets have a congruence relationship with the same dimensions as the datasets to which they apply. The association between a data product with its geolocation product is made on one of two ways: – The geolocation product may be packaged as a separate product group within the same HDF5 file. – The name of a separate geolocation product file may be stored in the N_GEO_Ref attribute on the root HDF group. – Choice of “as a product group” or “as a separate file” is made upon order from the NPOESS IDPS. NPOESS•INTEGRATED PROGRAM OFFICE 13 The HDF Workshop 6, 2007 Nov
Common Geolocation Fields for VIIRS Products Field Name Comments Dimensions Units Data Type StartTime since epoch 1/1/1958 [per scan or swath] microseconds 64-bit signed integer MidTime since epoch 1/1/1958 [per scan or swath] microseconds 64-bit signed integer SCPosition ECR coordinates [per scan or swath] meters SCVelocity ECR coordinates [per scan or swath] meters/second 32-bit float 32-bit float Latitude [per cell] degrees 32-bit float Longitude [per cell] degrees 32-bit float SolarZenithAngle [per cell] degrees 32-bit float SolarAzimuthAngle [per cell] degrees 32-bit float SensorZenithAngle [per cell] degrees 32-bit float SensorAzimuthAngle [per cell] degrees 32-bit float [per cell] meters 32-bit float [per cell] meters 32-bit float Height SatelliteRange geoid or terrain NPOESS•INTEGRATED PROGRAM OFFICE 14 The HDF Workshop 6, 2007 Nov
Product Profiles • XML documents provide definition of product fields. – Product Profile is delivered as part of the product documentation. – Contains metadata such as units of measure, dimension names, legend entries, etc – A separate profile per product, but each conforms to the same NPOESS Product document type definition (dtd) and XML schema definition (xsd). – A style sheet is provided that can render the profile for a web browser. – Example:VIIRI_IIT_II D RIxml NPOESS•INTEGRATED PROGRAM OFFICE 15 The HDF Workshop 6, 2007 Nov
17 field attributes in XML product profile (1..9) Attribute Name Type DataType String Description String Set of Boolean set of string String Set of string Set of number Set of string Set of number Dimension_GranuleBoundary Dimension_Name Field_Name FillValue_Name FillValue_Value LegendEntry_Name LegendEntry_Value Comments String format is: “%d-bit %s”, where %d is the number of bits and %s is one of: o signed integer o unsigned integer o floating point o <blank> (a bitfield) A descriptive text True (1) indicates that this is dimension extends when granules are appended. Name match indicates that this dimension is congruent with the same dimension names in other datasets in this product group. The name of the HDF5 dataset that contains the field values. Data type matches type of dataset. Data type matches type of dataset. NPOESS•INTEGRATED PROGRAM OFFICE 16 The HDF Workshop 6, 2007 Nov
17 field attributes in XML product profile (10..17) Attribute Name Type Comments MeasurementUnits String Consistent with SI naming and Unidata’s “udunits” package NumberOfDimensions Integer Integer greater than zero. NumberOfFillValues Integer NumberOfLegendEntries Integer RangeMax Number RangeMin Number Scaled Boolean ScaleFactorName String If zero, then no FillValue_Name and FillValue_Value attributes are present. Fill Values are used for primary data fields only. If zero, then no LegendEntry_Name and LegendEntry_Value attributes are present. Legend entries are used for quality fields only. Maximum expected value of field elements in the product, not just this dataset instance. Data type matches type of dataset. Minimum expected value of field elements in the product, not just this instance. Data type matches type of dataset. True indicates that the dataset is scaled. Note that fill values are in the dataset type and so must be tested before un-scaling. the name of the HDF5 dataset that contains scaling coefficients. To un-scale the elements, first multiply the scaled element by the first element and then add the second element. If the dataset is not scaled, Scale_AttributeName will not exist. NPOESS•INTEGRATED PROGRAM OFFICE 17 The HDF Workshop 6, 2007 Nov
The presentation will start after a short (15 second) video ad from one of our sponsors. Hot tip: Video ads won’t appear to registered users who are ...
HDF5 for NPP/JPSS Data Products. Richard E. UllmanNASA Goddard Space Flight CenterGreenbelt, MD. AMS • 2012. Michael J. Denning. Integrity Applications ...
Level 1 Data - Global (Sentinel-3) Title Resources Instrument; ... Product Data Format Specification - SLSTR Level 1 & Level 2 Instrument Products (PDF)
Product data management ... Product data management is part of product lifecycle management and configuration management, and is primarily used by engineers.
Enabling Rich Snippets for Products ... (in ISO 8601 date format) ... Microdata for Google Shopping describes further use of schema.org product data within ...
ISO 10303-21 defines the encoding mechanism on how to ... #14=PRODUCT_DEFINITION ... The time is given in the international data time format ISO ...
AP 236 (Furniture product data and project data)... ... wird auch das Format IFC genutzt. Siehe auch Industry Foundation Classes ...
... collaboration and data sharing across the product lifecycle. ... find solutions and learn more about the JT File Format and the JT Open Group. Learn More;
JPSS Common Data Format Control Book (CDFCB) - (Block/Revision 0123) Current Rev on MIS ... (OAD) for the CrIMMS Products Not in MIS - Product moved to NDE.