UML Representation of NPOESS Data Products in HDF5

0 %
100 %
Information about UML Representation of NPOESS Data Products in HDF5
Technology

Published on February 19, 2014

Author: HDFEOS

Source: slideshare.net

UML Representation of NPOESS Data Products in HDF5 Denise Dulaigh Chad Johnson William Johnsen, Ph.D. NPOESS Program Raytheon Company Aurora, Colorado

The National Polar-orbiting Operational Environmental Satellite System (NPOESS) MTG03-IDP-049 – Collects and distributes global satellite sensor data and creates NPOESS Data Products. – NPOESS Data Products are packaged into HDF5 files and delivered. – Delivered data can be read an any platform supported by HDF5. – There are a number of different types of data, discussed in further slides. – To meet Operational Latency requirements, the data is originally produced in granules (data blocks) that allow the system to operate efficiently. – A change in the size of the granules critically affects production efficiency. – Native metadata is associated with granules and aggregated granules. – Geolocation data in separate HDF5 files are referenced from NPOESS Data Product HDF5 files. HDF5 Workshop VII, 26 – 28 Oct 2004 2

NPOESS data – user view MTG03-IDP-049 How does our granule size affect users? – The size of the granules may not be the most desirable means to receive, analyze, store, or use the contained data. – The characteristics (metadata) of individual granules need to be retained for analysis and archival purposes. To increase usability: – NPOESS provides a means to package (aggregate) multiple granules into a single deliverable HDF5 file. – Metadata names are FGDC compliant. – For consistent access to data in the produced HDF5 files, access to data uses hyperslab references. HDF5 Workshop VII, 26 – 28 Oct 2004 3

NPOESS data types MTG03-IDP-049 – Raw Data Records (RDRs) – Full resolution, unprocessed time-referenced digital sensor data and other data required to compute SDRs. – Temperature Data Records (TDRs) – Geolocated, antenna temperatures with all relevant calibration data counts and ephemeris data, from passive microwave sensors only. – Sensor Data Records (SDRs) – Computed from RDRs; full resolution sensor data that are time referenced, earth located, and calibrated. – Environment Data Records (EDRs) – Data records that contain the environmental parameters or imagery required to be generated as user products, produced by applying an appropriate set of algorithms to SDRs. – Deliverable Intermediate Products (IPs) – Data records that are produced by applying an appropriate set of algorithms to SDRs, but is not on the official list of deliverable EDRs. Currently, Cloud Mask is the only deliverable IP. – Selected data, depending on the user: auxiliary data, ancillary data. – The UML diagram, data structure, and data types of all NPOESS HDF5 data is documented in the Common Data Format Control Book – External (CDFCB-X) ** Formal definitions (not operational) from the NPOESS Glossary HDF5 Workshop VII, 26 – 28 Oct 2004 4

Schematic of an HDF5 Dataset MTG03-IDP-049 A multidimensional array of data elements Dataset Header with metadata • • • • Dataspace (intrinsic) Datatype (intrinsic) Storage layout Attributes Header Data start time = 32.4 Dataspace data type = ‘SDR’ int16 Datatype 2 Rank Dim_3=2 algorithm = ‘1.1’ Attributes Dim_2=4 Dim_1=5 Dimensions HDF5 Workshop VII, 26 – 28 Oct 2004 Array Datatype Chunked; compressed Storage layout 5

Schematic of an HDF5 File Aggregated Granule MTG03-IDP-049 Root File Secondary Data Attributes Root Attributes Secondary Data Data Product Data Product Attributes Data Granule 1 Granule N Array Datatype Secondary Data Aggregated Granules Array Datatype Array Datatype Data Product Data Ref 1 Data Ref 2 Data Ref 3 HDF5 Workshop VII, 26 – 28 Oct 2004 * Data arrays grow in 1 direction only 6

Schematic of an HDF5 File Granule 1 MTG03-IDP-049 Root File Secondary Data Attributes Root Attributes Secondary Data Data Product Data Product Attributes Data Granule 1 Granule N Array Datatype Secondary Data Aggregated Granules Array Datatype Array Datatype Data Product Data Ref 1 Data Ref 2 Data Ref 3 HDF5 Workshop VII, 26 – 28 Oct 2004 * Data arrays grow in 1 direction only 7

UML CASE tools to describe HDF5 file layout MTG03-IDP-049 – To more efficiently communicate and better control our HDF5 structures, we use Computer-Aided Software Engineering (CASE) tools to design the structure. – Since HDF5 data structures have object-oriented characteristics, we use Unified Modeling Language (UML) to describe the nature of our HDF file structure. – HDF5 groups, datasets, and array datatypes are UML classes; HDF5 attributes and references are UML attributes. Stereotypes are used to differentiate HDF5 objects. – UML class derivation (HDF5 group inclusion) and multiplicity (HDF5 file structural relationships) give the user an idea about what to expect before viewing the HDF5 file. – Data is stored in a separate HDF5 group (Data group). – Each array datatype contains one or more hyperslab references to a subset of data in the Data group. HDF5 Workshop VII, 26 – 28 Oct 2004 8

Schematic to UML mapping MTG03-IDP-049 File Root <<H_G>> HDF_EDR_Product Group (f rom H DF_File_Root) <<Root>> HDF_File_Root Mission_Name : H_STR N_Quas i-Static_Metadata_Reference Identifier_Information.Dataset_Indentifier : H_STR Attributes 1 Band_Measurement_Mode_ID : H_STR Instrument_Short_Name : H_STR N_Dataset_Type_Tag : H_STR N_Instrument_Flight_SW_Version : H_STR N_Spacecraft_SW_Vers ion : H_STR N_Subs ystem _SW_Version : H_STR Platform_Short_Name : H_STR Processing_Level_Identifier : H_STR 1..n <<H_DS>> HDF_EDR_Product_Granule_n Dataset (f rom HDF_ED R_Produc t Group) Data Product Attributes Dataset Attributes Ascending/Decending_Indicator : H_STR Beginning_Date : H_CU Beginning_Time : H_CU Cloud_Cover : H_I East_Bounding_Coordinate : H_F Ending_Date : H_CU Ending_Time : H_CU G_Ring_Latitude : H_F Array G-Ring_Longitude : H_F Array Input_Dataset_Identifier.Dataset_Identifier : H_STR N_Algorithm_Vers ion : H_STR N_Anc_Type_Used : N_STR N_Automatic_Quality_Parameter_Value : H_F N_Automatic_Quality_Test_Explanation : H_STR N_Beginning_Orbit_Number : H_UI N_Beginning_Tim e_IET : H_LL N_Creation_Date : H_STR N_Creation_Tim e : H_STR N_Day_Night_Flag : H_F N_Degraded_Mode : H_STR N_EDR_Field : H_F N_EDR_Scale_Factor : H_F N_Ending_Time_IET : H_LL N_Graceful_Degradation : H_STR N_Granule_ID : H_I N_Granule_Version : H_STR N_LEO_Flag : H_STR N_Nadir_Latitude_Max : H_F N_Nadir_Latitude_Min : H_F N_Nadir_Longitude_Max : H_F N_Nadir_Longitude_Min : H_F N_Percent_Miss ing_Data : H_F N_Solar_Zenith_Angle_Max : H_F N_Solar_Zenith_Angle_Min : H_F N_Update_Date : H_STR N_Update_Time : H_STR North_Bounding_Coordinate : H_F Operational_Mode : H_STR South_Bounding_Cooridinate : H_F West_Bounding_Coordinate : H_F Datatype 1 <<H_RR Array>> (EDR VIIRS SST) (f rom HDF_EDR_Product_Gr anule_n Dataset) Data Ref Group – HDF structural element Skin_Sea_Surface_Tempurature Ref : H_RR Bulk_Sea_Surface_Tempurature Ref : H_RR Quality_Flags Ref : H_RR * Data types are explained on chart 15-16 Dataset – a Group that also contains a header and data Datatype – internal to HDF, used by datasets to reference read functions, included in Dataset Attributes – where the NPOESS metadata is stored Data Ref – hyperslab reference to Data group HDF5 Workshop VII, 26 – 28 Oct 2004 9

Example UML HDF File Structure (EDR) MTG03-IDP-049 <<H_G>> HDF_EDR_Product Group (f rom HDF_File_Root) <<Root>> HDF_File_Root Mission_Name : H_STR N_Quasi-Static_Metadata_Reference Identifier_Information.Dataset_Indentifier : H_STR Band_Measurement_Mode_ID : H_STR Instrument_Short_Name : H_STR N_Dataset_Type_Tag : H_STR N_Instrument_Flight_SW_Version : H_STR N_Spacecraft_SW_Version : H_STR N_Subsystem _SW_Version : H_STR Platform_Short_Nam e : H_STR Processing_Level_Identifier : H_STR 1 1..n • There is a single primary Product Group and zero or more secondary Groups of supporting data. • Each Product Group is accessed from the <<Root>> HDF element. <<H_DS>> HDF_EDR_Product_Granule_n Dataset (f rom HDF_EDR_Product Group) Ascending/Decending_Indicator : H_STR Beginning_Date : H_CU Beginning_Time : H_CU Cloud_Cover : H_I East_Bounding_Coordinate : H_F Ending_Date : H_CU Ending_Time : H_CU G_Ring_Latitude : H_F Array G-Ring_Longitude : H_F Array Input_Dataset_Identifier.Dataset_Identifier : H_STR N_Algorithm _Version : H_STR N_Anc_Type_Used : N_STR N_Automatic_Quality_Parameter_Value : H_F N_Automatic_Quality_Test_Explanation : H_STR N_Beginning_Orbit_Number : H_UI N_Beginning_Time_IET : H_LL N_Creation_Date : H_STR N_Creation_Time : H_STR N_Day_Night_Flag : H_F N_Degraded_Mode : H_STR N_EDR_Field : H_F N_EDR_Scale_Factor : H_F N_Ending_Time_IET : H_LL N_Graceful_Degradation : H_STR N_Granule_ID : H_I N_Granule_Version : H_STR N_LEO_Flag : H_STR N_Nadir_Latitude_Max : H_F N_Nadir_Latitude_Min : H_F N_Nadir_Longitude_Max : H_F N_Nadir_Longitude_Min : H_F N_Percent_Missing_Data : H_F N_Solar_Zenith_Angle_Max : H_F N_Solar_Zenith_Angle_Min : H_F N_Update_Date : H_STR N_Update_Time : H_STR North_Bounding_Coordinate : H_F Operational_Mode : H_STR South_Bounding_Cooridinate : H_F West_Bounding_Coordinate : H_F * Data types are explained on chart 15-16 1 <<H_RR Array>> (EDR VIIRS SST) (f rom H DF_ED R_Product_Granule_n Dataset) Skin_Sea_Surface_Tempurature Ref : H_RR Bulk_Sea_Surface_Tempurature Ref : H_RR Quality_Flags Ref : H_RR HDF5 Workshop VII, 26 – 28 Oct 2004 • Multiple Granule datasets may be provided per Product Group, depending on data requested from NPOESS. An aggregation dataset will always exist regardless of the number of Granules. The aggregation dataset will contain aggregated attributes that characterize the set of all Granules. • Each Granule dataset as well as the Aggregation dataset contain a single Array Datatype. • The Array Datatype contains hyperslab references to the Data group. 10

Example Complete HDF5 File Structure (EDR) Root Group MTG03-IDP-049 (f rom HDF_EDR_Product Group) Mission_Name : H_STR N_Quasi-Static_Metadata_Reference Identifier_Information.Dataset_Indentifier : H_STR Data Product Group Aggregate Dataset <<H_DS>> HDF_EDR_Product_Aggregation Dataset <<Root>> HDF_File_Root 1 <<NPOESS>> AggregateBeginningDate : H_CU <<NPOESS>> AggregateBeginningGranuleID : H_IL <<NPOESS>> AggregateBeginningOrbitNumber : H_IL <<NPOESS>> AggregateBeginningTime : H_CU <<NPOESS>> AggregateCreationDate : H_STR <<NPOESS>> AggregateCreationTime : H_STR <<NPOESS>> AggregateEndingDate : H_CU <<NPOESS>> AggregateEndingGranuleID : H_IL <<NPOESS>> AggregateEndingOrbitNumber : H_IL <<NPOESS>> AggregateEndingTime : H_CU <<NPOESS>> AggregateG-RingLatitude : H_F Array <<NPOESS>> AggregateG-RingLongitude : H_F Array <<NPOESS>> AggregatePercentMissingDataAve : H_F <<H_RO Array>> (EDR VIIRS SST Aggregation) (f rom HDF _EDR_Product_Aggregation Dataset) 1 Skin_Sea_Surface_Tempurature Ref : H_RO Quality_Flags Ref : H_RO Bulk_Sea_Surface_Tempurature Ref : H_RO 1 <<H_G>> HDF_EDR_Product Group (f rom HDF_File_Root) Band_Measurement_Mode_ID : H_STR Instrument_Short_Name : H_STR N_Dataset_Type_Tag : H_STR N_Instrument_Flight_SW_Version : H_STR N_Spacecraft_SW_Version : H_STR N_Subsystem_SW_Version : H_STR Platform_Short_Name : H_STR Processing_Level_Identifier : H_STR <<H_DS>> HDF_EDR_Product_Granule_n Dataset (f rom HDF_EDR_Product Group) 1..n Data Group Hidden from view Secondary Group As applicable HDF5 Workshop VII, 26 – 28 Oct 2004 Ascending/Decending_Indicator : H_STR Beginning_Date : H_CU Beginning_Time : H_CU Cloud_Cover : H_I East_Bounding_Coordinate : H_F Ending_Date : H_CU Ending_Time : H_CU G_Ring_Latitude : H_F Array G-Ring_Longitude : H_F Array Input_Dataset_Identifier.Dataset_Identifier : H_STR N_Algorithm_Version : H_STR N_Anc_Type_Used : N_STR N_Automatic_Quality_Parameter_Value : H_F N_Automatic_Quality_Test_Explanation : H_STR N_Beginning_Orbit_Number : H_UI N_Beginning_Time_IET : H_LL N_Creation_Date : H_STR N_Creation_Time : H_STR N_Day_Night_Flag : H_F N_Degraded_Mode : H_STR N_EDR_Field : H_F N_EDR_Scale_Factor : H_F N_Ending_Time_IET : H_LL N_Graceful_Degradation : H_STR N_Granule_ID : H_I N_Granule_Version : H_STR N_LEO_Flag : H_STR N_Nadir_Latitude_Max : H_F N_Nadir_Latitude_Min : H_F N_Nadir_Longitude_Max : H_F N_Nadir_Longitude_Min : H_F N_Percent_Missing_Data : H_F N_Solar_Zenith_Angle_Max : H_F N_Solar_Zenith_Angle_Min : H_F N_Update_Date : H_STR N_Update_Time : H_STR North_Bounding_Coordinate : H_F Operational_Mode : H_STR South_Bounding_Cooridinate : H_F West_Bounding_Coordinate : H_F <<H_RR Array>> (EDR VIIRS SST) 1 (f rom HDF_EDR_Product_Granule_n Dataset) Skin_Sea_Surface_Tempurature Ref : H_RR Bulk_Sea_Surface_Tempurature Ref : H_RR Quality_Flags Ref : H_RR Granule Dataset * Data types are explained on chart 15-16 11

1 <<NPOESS>> AggregateG-RingLatitude : H_F Array <<NPOESS>> AggregateG-RingLongitude : H_F Array <<NPOESS>> AggregatePercentMissingDataAve : H_F Example Complete HDF5 File Structure (EDR) Granule Dataset MTG03-IDP-049 <<H_DS>> HDF_EDR_Product_Granule_n Dataset TR (f rom HDF_EDR_Product Group) 1..n Ascending/Decending_Indicator : H_STR Beginning_Date : H_CU Beginning_Time : H_CU Cloud_Cover : H_I East_Bounding_Coordinate : H_F Ending_Date : H_CU Ending_Time : H_CU G_Ring_Latitude : H_F Array G-Ring_Longitude : H_F Array Input_Dataset_Identifier.Dataset_Identifier : H_STR N_Algorithm_Version : H_STR N_Anc_Type_Used : N_STR N_Automatic_Quality_Parameter_Value : H_F N_Automatic_Quality_Test_Explanation : H_STR N_Beginning_Orbit_Number : H_UI N_Beginning_Time_IET : H_LL N_Creation_Date : H_STR N_Creation_Time : H_STR N_Day_Night_Flag : H_F N_Degraded_Mode : H_STR N_EDR_Field : H_F N_EDR_Scale_Factor : H_F N_Ending_Time_IET : H_LL N_Graceful_Degradation : H_STR N_Granule_ID : H_I N_Granule_Version : H_STR N_LEO_Flag : H_STR N_Nadir_Latitude_Max : H_F N_Nadir_Latitude_Min : H_F N_Nadir_Longitude_Max : H_F N_Nadir_Longitude_Min : H_F N_Percent_Missing_Data : H_F N_Solar_Zenith_Angle_Max : H_F N_Solar_Zenith_Angle_Min : H_F N_Update_Date : H_STR N_Update_Time : H_STR North_Bounding_Coordinate : H_F Operational_Mode : H_STR South_Bounding_Cooridinate : H_F West_Bounding_Coordinate : H_F HDF5 Workshop VII, 26 – 28 Oct 2004 <<H_RR Array>> (EDR VIIRS SST) 1 (f rom HDF_EDR_Product_Granule_n Dataset) Skin_Sea_Surface_Tempurature Ref : H_RR Bulk_Sea_Surface_Tempurature Ref : H_RR Quality_Flags Ref : H_RR * Data types are explained on chart 15-16 12

HDF5 file structure – Example CDFCBX Table for RDR Product Granule Group (primary data) HDF5 Workshop VII, 26 – 28 Oct 2004 * Data types are explained on chart 15-16 MTG03-IDP-049 13

HDF5 file structure – Example CDFCBX Table for RDR Auxiliary Data Calibration Coefficients Group (secondary data) MTG03-IDP-049 * Data types are explained on chart 15-16 HDF5 Workshop VII, 26 – 28 Oct 2004 14

HDF5 file structure – HDF5 data type to CDFCB-X cross-reference HDF5 Workshop VII, 26 – 28 Oct 2004 MTG03-IDP-049 15

HDF5 file structure – HDF5 attribute to CDFCB-X cross-reference HDF5 Workshop VII, 26 – 28 Oct 2004 MTG03-IDP-049 16

Outstanding considerations MTG03-IDP-049 – Chunking for SDRs and EDRs – Compression functions and analysis of NPOESS data records – Storage of bit-meaningful data in bit-field data types HDF5 Workshop VII, 26 – 28 Oct 2004 17

Questions/Comments MTG03-IDP-049 HDF5 Workshop VII, 26 – 28 Oct 2004 18

Add a comment

Related presentations

Related pages

UML Representation of NPOESS Data Products in HDF5 ...

UML Representation of NPOESS Data Products in HDF5. Denise Dulaigh Chad Johnson William Johnsen, Ph.D. NPOESS Program Raytheon Company Aurora, Colorado.
Read more

UML Representation of NPOESS Data Products in HDF5

UML Representation of NPOESS Data Products in HDF5. Denise Dulaigh Chad Johnson William Johnsen, Ph.D. NPOESS Program Raytheon Company Aurora, Colorado.
Read more

UML Representation of NPOESS Data Products in HDF5 ...

The document was removed. Please view another documents! 18 × Close Share UML Representation of NPOESS Data Products in HDF5
Read more

P2.10 "UML Representation of NPOESS Data Products in HDF5 ...

GOES-R/NPOESS Symposium. P2.10 "UML Representation of NPOESS Data Products in HDF5" William Johnsen, Raytheon Company, Aurora, CO; and D. Dulaigh and C ...
Read more

HDF5 for NPOESS Data Products - Technology

1.HDF5 for NPOESS Data Products Alan M. Goldberg The MITRE Corporation agoldber@mitre.orgThis work was ... UML Representation of NPOESS Data Products in HDF5.
Read more

Agenda - HDF and HDF-EOS Workshop VIII

HDF and HDF-EOS Workshop VIII. ... UML Representation of NPOESS Data Products in HDF5: n/a: ... Using HDF5 to Store Engineering Test Data ...
Read more

PowerPoint Presentation - HDF-EOS

UML Representation of NPOESS Data Products in HDF5 Denise Dulaigh Chad Johnson William Johnsen, Ph.D. NPOESS Program Raytheon Company Aurora, Colorado
Read more

A Proposal to Provide HDF Support for - The HDF Group

HDF Support for the ... NPOESS products and applications to use HDF5 in ... on the NPOESS data representation in HDF5 using UML and ...
Read more

PPT – NPP NPOESS Product Profile of HDF5 PowerPoint ...

... products are constructed using the ... NPP NPOESS Product Profile of HDF5. ... The association between a data product with its geolocation product is ...
Read more