Profile of NPOESS HDF5 Files

50 %
50 %
Information about Profile of NPOESS HDF5 Files
Technology

Published on February 19, 2014

Author: HDFEOS

Source: slideshare.net

Description

The NPOESS program uses Unified Modeling Language (UML) to describe the format of the HDF5 files produced. For each unique type of data product, the HDF5 storage organization and the means to retrieve the data is the same. This provides a consistent data retrieval interface for manual and automated users of the data, without which would require custom development and cumbersome maintenance. The data formats are described using UML to provide a profile of HDF5 files.

Profile of NPOESS HDF5 Files Kim Tomashosky, Ken Stone, Pat Purcell, Ron Andrews NPOESS Program Raytheon Company Aurora, Colorado

NPOESS HDF5 General Overview Ron Andrews

HDF5 Conceptual Diagram XML User’s Block Product Group Root - / Agg Reference Object Data Gran n Reference Region Reference Region 3

HDF5 XML User’s Block • • The XML User’s Block for NPOESS Data Products provides a ‘quick-look’ into the metadata of the associated HDF5 file. The XML User’s Blocks are defined in the following volumes: – CDFCB-X Volume V – contains the XML User’s Block formats for the RDRs, SDRs, TDRs, EDRs, ARPs, and IPs – CDFCB-X Volume VI – Ancillary, Auxiliary, Reports, and Messages – contains the XML User’s Block formats for the ancillary, auxiliary, reports, and messages that are wrapped in HDF5 – The size of the HDF5 XML User’s Block will be a factor of 512 bytes, with a maximum size which is still to be determined • Provides elements: – – – – – – – – N_Processing_Domain Mission, Platform, and Instrument Names N_Dataset_Type_Tag Number_of_Data_Products CollectionShortName(s) Aggregation Information Timestamps (Creation Timestamp, Observation Timestamps) Percent Missing Data 4

General HDF5 File Structure Root Group Attributes are here. <<Root>> NPOESS General HDF_File_Root A root group may have one or more data products. A Data Product Group may have one or more product data Product groups contain datasets for aggregations and individual granules. <<H_G>> <<H_G>> HDF_xxx_Product <<H_G>> HDF_xxx_Product_Data <<H_G>> HDF_All_Data data for a given NPOESS data product <<H_DS>> xxx_Data (from NPOESS General HDF_File_Root) ... ) <<H_DS>> 1...* HDF_xxx_Product_Granule_# (from HDF_xxx_Product) (from HDF_xxx_Product) AggregateDataset : H_RO (from HDF_xxx_Product_Aggregation) ... ) The xxx_Product_Data contains a collection of product datasets Data Product <<H_DS>> HDF_xxx_Product_Aggregation <<H_RO Array>> AggregateDatasetArray The All_Data group is populated with nested xxx_Product_Data groups. Dataset : H_RR The Product Aggregation contains a regional object (H_RO) to the data. xxx is the product type, for example RDR, SDR, EDR, etc. # is a number corresponding to the granule number within the aggregated dataset. <<H_RR Array>> DatasetArray (from HDF_xxx_Product_Granule_#) ... ) The Product Granule contains a regional reference (H_RR) to the data. xxx is the product type, for example RDR, SDR, EDR, etc. # is a number corresponding to the granule number within the aggregated dataset. The dataset is a regional reference (H_RR) to the data. 5

HDF5 File Root Group Metadata Elements – Example CDFCB-X Table ANC AUX Gridded IP GEO IP/ARP/EDR SDR/TDR RDR X X X X Field Name Data Type Calendar_Date H_STR Identification_Information.Dataset_Identifier H_STR X X X X Instrument_Short_Name H_STR X X X X X Mission_Name H_STR X X X X X X X N_Creation_Date H_STR X X X X X X X N_Creation_Time H_STR X X X X N_Dataset_Source H_STR X X X X X X X N_Geo_Ref H_STR X X X X X X X N_NPOESS_Document_Ref H_STR X X X X N_Orbit_Position_Predicted (NPOESS only) H_STR X X N_Subsystem_SW_Version H_STR N_Update_Date H_STR X N_Update_Time H_STR X Platform_Short_Name H_STR Time_Of_Day H_STR X X X X X X X X X X X X X 6

HDF5 File Product Group Metadata Elements – Example CDFCB-X Table AUX ANC Gridded IP GEO IP/ARP/EDR SDR/TDR RDR Field Name Data Type Band_ID H_STR X Band_Measurement_Mode_ID H_STR X N_Anc_Type_Tasked H_STR N_Collection_Short_Name H_STR X X X X X X X N_Dataset_Type_Tag H_STR X X X X X X X N_Instrument_Flight_SW_Version H_I X X N_Packet_Type H_STR X N_Packet_Type_Count H_STR X N_Processing_Domain H_STR X X X X X X X N_Spacecraft_SW_Version H_STR X X X Operational_Mode H_STR X X Processing_Level_Identifier H_STR X X X X X X X X 7

HDF5 File Product Aggregation Dataset Metadata Elements – Example CDFCB-X Table IP/ARP/EDR GEO H_STR X X X X AggregateBeginningGranuleID H_STR X X X AggregateBeginningOrbitNumber H_ILLU X X X AggregateBeginningTime H_STR X X X AggregateEndingDate H_STR X X X AggregateEndingGranuleID H_STR X X X AggregateEndingOrbitNumber H_ILLU X X X AggregateEndingTime H_STR X X X X AggregateG-RingLatitude H_F Array X X X AggregateG-RingLongitude H_F Array X X X AggregateNumberGranules H_F X X X X AggregatePercentMissingDataAve H_F X X X AUX SDR/TDR AggregateBeginningDate ANC Data Type Gridded IP RDR Field Name X X X X X 8

HDF5 File Product Granule Dataset Metadata Elements – Example CDFCBX Table X X X X X X X Ascending/Descending_Indicator H_C Beginning_Date H_STR Beginning_Time H_STR Cloud_Cover H_F East_Bounding_Coordinate H_STR X X X Ending_Date H_STR X X X H_STR X X X X G-Ring_Latitude H_F Array X X X X G-Ring_Longitude H_F Array X X X X Input_Dataset_Identifier.Dataset_Identifier H_STR X X X N_Algorithm_Version H_STR X X X N_Anc_Filename H_STR X X N_Anc_Type H_STR N_Aux_Filename H_STR X X X N_Beginning_Orbit_Number H_ILLU X X X N_Beginning_Time_IET H_ILL X X X X X Ending_Time AUX X Data Type ANC X X Gridded IP IP/ARP/EDR GEO SDR/TDR RDR X Field Name X X X X X X X X 9

HDF5 File Product Granule Dataset Metadata Elements – Example CDFCBX Table X X X N_Day_Night_Flag H_STR X X X N_Deliverable_IP_Filename H_STR X X N_Ending_Time_IET H_ILL X X N_Graceful_Degradation H_STR N_Granule_ID H_STR N_Granule_Quality_Value H_F N_Granule_Version H_STR N_LEOA_Flag H_STR N_Levels X AUX GEO H_IL ANC IP/ARP/EDR N_Columns Gridded IP Data Type SDR/TDR RDR Field Name X X X X X X X X X X X X X X X X X X X X H_STR X X N_LUT_Version H_STR X X N_Nadir_Latitude_Max H_F X X X N_Nadir_Latitude_Min H_F X X X N_Nadir_Longitude_Max H_F X X X N_Nadir_Longitude_Min H_F X X X N_Percent_Erroneous_Data H_F X X 10

HDF5 File Product Granule Dataset Metadata Elements – Example CDFCBX Table H_F N_Percent_Not-Applicable_Data H_F X N_Recent_Maneuver H_F X N_Reference_ID H_STR X X N_Rows H_STR X X X N_Satellite/Local_Azimuth_Angle_Max H_IL X X X N_Satellite/Local_Azimuth_Angle_Min H_F X X N_Satellite/Local_Zenith_Angle_Max H_F X X N_Satellite/Local_Zenith_Angle_Min H_F X X N_Solar_Azimuth_Angle_Max H_F X X N_Solar_Azimuth_Angle_Min H_F X X N_Solar_Zenith_Angle_Max H_F X X N_Solar_Zenith_Angle_Min H_F X X N_Spacecraft_Manuever H_F X X N_Spacecraft_SW_Version H_STR X X N_Tile_ID H_STR X X AUX N_Percent_Missing_Data ANC IP/ARP/EDR X Data Type Gridded IP SDR/TDR X GEO RDR X Field Name X X 11

HDF5 File Product Granule Dataset Metadata Elements – Example CDFCBX Table H_I X X X Processing_Level_Identifier H_F X X South_Bounding_Coordinate H_STR X X X X West_Bounding_Coordinate H_STR X X X AUX North_Bounding_Coordinate ANC IP/ARP/EDR Gridded IP Data Type SDR/TDR GEO RDR Field Name X X X 12

HDF5 File Spacecraft Diary Group Metadata Elements – Example CDFCBX Table X N_Packet_Type H_STR X N_Packet_Type_Count H_STR X Processing_Level_Identifier H_STR AUX H_STR ANC X N_Dataset_Type_Tag Gridded IP H_STR GEO N_Collection_Short_Name IP/ARP/EDR Data Type SDR/TDR RDR Field Name X 13

HDF5 File Spacecraft Diary Aggregation Metadata Elements – Example CDFCB-X Table X AggregateEndingDate H_STR X AggregateEndingTime H_STR AUX H_STR ANC X AggregateBeginningTime Gridded IP H_STR GEO AggregateBeginningDate IP/ARP/EDR Data Type SDR/TDR RDR Field Name X 14

HDF5 File Spacecraft Diary Dataset Metadata Elements – Example CDFCBX Table X N_Ending_Time_IET H_ILL X N_Support_Data_Version H_STR AUX H_ILL ANC X N_Beginning_Time_IET Gridded IP H_StR GEO Input_Dataset_Identifier.Dataset_Identifier IP/ARP/EDR Data Type SDR/TDR RDR Field Name X 15

HDF5 Conceptual Diagram - Data t y x H_RR H_RO 256x3200xN 256x3200 16

NPOESS Quality Flags Overview • • • The concept is to provide for consistently stored, high density, quality information about the delivered data – simplifying usability while maintaining storage efficiency Quality flags are qualifications of one or more consecutive bits in each byte. Quality flag arrays follow the structure of the data product – The size of the arrays are equal to or less than the size of the data to which the quality information applies (dimensions retain correspondence to data product arrays) • Quality flags will appear in the HDF5 files as n number(s) of two or three dimensional, 1-byte arrays. – The number of arrays is dependant on the number and size of each data product's quality flag definitions – Quality flags will be packed such that each flag is entirely contained within a single byte, occasionally resulting in a byte with reserved or meaningless bits – Byte alignment is the same for every quality flag array • First bit is the LSB, bytes are Big-Endian • Each byte may contain multiple bit-level flags 17

2-Dimensional Array Example Array #3 Array #3 Columns Array #2 Array #2 Data Array #1 Rows Array #1 18

3-Dimensional Array Example Layers Layers Array #2 Array #2 Columns Layers Layers Data Array #1 Rows Array #1 19

Detailed NPOESS UML Models Ken Stone

RDR UML Model < <R oo t>> / (RD R) < < H_ G> > A l l_ Da ta I d en ti f i er_ In form a ti on .D at a set _I n de nt i fi er : H_S T R I n st rum en t _S ho rt_ Na m e : H_S TR M issio n _N am e : H_S TR N _C re a ti o n_ Da te : H_ ST R N _C re a ti o n_ T i m e : H_S T R N _D at a se t _S ou rce : H _S TR N _D ocum en t _R ef : H _S TR N _P ro cessin g _D om a i n : H_ ST R N _R ef e re n ce _I D : H_ ST R N _S ub syst em _S W_V e rsi on : H_S T R N _U pd ate _Date : H_ S T R N _U pd ate _T i m e : H_ ST R P la tf orm _ Sh ort_ Na m e : H_ ST R 1 1 . .n << H_ G> > < Da t a P ro d uct C ol l ect i on Sh ort Nam e> _A l l (from All_D ata) 1 ..n << H_ D S>> D at a se t_A rra y (from <D ata Product Collection Short Name> _ All) < <H_G >> S PA CE CR AFT -DIA RY -R DR N_ Co l l ect io n_ Sh o rt _ Na m e : H_ S TR N_ Da t ase t_ T yp e_ Ta g : H_ ST R N_ Pa cke t _T yp e : H_ ST R N_ Sp a ce cra f t_ SW _V ersi on : H_ S TR Pro ce ssi n g_ Le vel _Id en tifi e r : H_ ST R N_ Nu m _ Gra nu l es : H_I 1 << H_ DS> > <S C Di a r Co l lec ti on S ho rt Nam e >_ Gran_ n y Ca l en da r_Date : H _S TR I np ut _ Da t aset_ Ide nt i fie r.D at a se t_I d en t if i e r : H_ ST R N_ B eg in n in g_ T i m e_ IE T : H_ I LL N_ E nd in g _T im e_ I ET : H_ I LL N_ S up po rt_ Da t a_ Ve rsi o n : H_ ST R T i m e_ of _ Da y : H_ ST R Da tase t A rra y : H_ RO 1 1 < <H _G >> Da t a_ Pro duct s << H _G >> < RD R Co l lect io n S ho rt Nam e > 1 1 1 << H_ G> > <S en so r> S DR C al i bra tio n C oe ffi ci en ts N_Col l e cti on _S ho rt _N am e : H _S TR N_Dat a set_T ype _T ag : H _S T R N_P ac ket_ Type : H _S T R N_S pa cecraf t_S W_ Ve rsi o n : H _S TR P rocessi ng _L eve l_ Ide nti f ie r : H_S TR N_Num _G ran ul e s : H_ I (from <RDR Collection Short Name>) 1..n 1 << H_ DS >> <S C D ia ry C oll ecti on Sho rt Nam e>_A gg N_ Co ll e ct i on _ Sh ort _N am e : H_ ST R N_ Da ta set_ Type _T ag : H _S TR N_ Pa cket_ Typ e : H _S T R N_ Sp acecraf t_SW _ Ve rsi o n : H_ ST R Proce ssi ng _L e ve l_ I de nti f i er : H_ST R N_ Nu m _G ran ul e s : H_ I 1 A gg reg ate Be gi n ni ng Date : H_ ST R A gg reg ate Be gi n ni ng T im e : H_ ST R A gg reg ate En di n gD ate : H_ ST R A gg reg ate En di n gT im e : H_ ST R D ata se t A rray : H_R O Ag greg a te Be g in ni n gDat e : H_S TR Ag greg a te Be g in ni n gG ran ul e ID : H _S TR Ag greg a te Be g in ni n gO rbi t Nu m b er : H _IL LU Ag greg a te Be g in ni n gT i m e : H_ ST R Ag greg a te Cre ati on Da te : H_ ST R Ag greg a te Cre ati on Ti m e : H _ STR Ag greg a te En d in gDat e : H _S TR Ag greg a te En d in gG ran ul e ID : H _S T R Ag greg a te En d in gO rbi t Nu m b er : H _I L LU Ag greg a te En d in gT i m e : H_ ST R Ag greg a te Pe rce n tM issi ng Da taA ve : H _F Da ta set Array : H_R O << H_ DS >> <RDR Col l e ct i on Sh ort Na m e >_ Gran _n B eg i nn in g _Dat e : H_ ST R B eg i nn in g _T im e : H_ ST R E nd i ng _D at e : H_ ST R E nd i ng _T im e : H_ ST R N _B eg i nn in g _O rb i t_ Nu m b er : H_ IL L U = N O N _B eg i nn in g _T im e_ I ET : H_ I LL N _E nd i ng _T i m e_ IET : H_ILL N _G ra n ul e_ I D : H_ IL L N _G ra n ul e_ Ve rsi o n : H_ ST R N _L EO _F l ag : H_ S T R N _P acket _T ype _C ou nt : H_ ST R N _P ercen t_Mi ssin g _Dat a : H_ F D ata se t A rra y : H_ RO 1. .n << H_ DS > > <se nso r S DR ca li b ra ti on co eff ici en ts> _ Gran _n (from <Sensor> SDR Calibration Coefficients) C al end ar _ Da t e : H_ ST R In pu t_Da t ase t _Ide n ti f i er.Da t ase t_ Ide nt i fi e r : H_S T R N _Beg i nni ng _ Ti m e _ IET : H _IL L N _End i ng_T i me _IET : H_I L L N _Sup port _D at a _V ersio n : H _S T R T i me _o f_ Da y : H_ S T R < <H DF >> D a t ase t Arra y : H_ RO < <H _D S> > < R DR Co l l ec t ion S ho rt N am e > _Ag g 1 .. n 1 < <H_D S> > <sen sor SDR cal i bra ti o n coe ffi ci en ts> _A gg (from <Sensor> SDR Calibration Coefficients) << NP OE SS> > A gg reg ate Be gi n ni n gD ate : H_ ST R << NP OE SS> > A gg reg ate Be gi n ni n gT im e : H_ ST R << NP OE SS> > A gg reg ate En di n gDat e : H_ ST R << NP OE SS> > A gg reg ate En di n gT i m e : H_ ST R Da ta set Array : H _R O 21

Common RDR Layout 22

SDR/TDR UML Model 23

EDR UML Model <<H_G> >All_Data 1 1 1 1.. n <<H_G> > <Data Product Collection Short Name>_All (from All_Data) N_GEO_Ref and inclusion of the Geolocation Group is dependant on the Packaging Option configured on the IDP. 1 N_Colection_Short_Name : H_STR N_Dataset_Type_Tag : H_STR N_Processing_Domain : H_STR N_ Num_Granules : H_I Operational_Mode : H_STR Processing_Level_Identifier : H_STR <<H_DS> > Dataset_Arra 1 1 1..n 1 1 1..n 1 1 (from <Data Product Collection Short Name>_All) y N_Columns : H_ IU N_Rows : H_IU Ascending/Descending Indicator : H_C Beginning_Date : H_STR Beginning_Time : H_STR East_Bounding_Coordinate : H_F Ending_Date : H_STR Ending_Time : H_STR G-Ring_Latitude : H_F G-Ring_Longitude : H_F N_Algorithm_Version : H_STR N_Beginning_Orbit_Number : H_ILLU N_Beginning_Time_IET : H_ILL N_Day_Night_FLAG : H_STR N_Ending_Time_IET : H_ILL N_Granule_ID : H_ILL N_Granule_Quality_Value : H_F N_Granule_Quality_Value_Explanation : H_STR N_Granule_Version : H_STR N_LEO_Flag : H_STR N_ Nadir_Latitude_Max : H_F N_ Nadir_Latitude_Min : H_F N_ Nadir_Longitude_Max : H_F N_ Nadir_Longitude_Min : H_F N_Reference_ID : H_STR North_Bounding_Coordinate : H_F South_Bounding_Coordinate : H_F West_Bounding_Coordinate : H_F Dataset Array : H_RR 1 <<H_G> > Data_Product s 1 <<H_G>> <GEO Collection Short Name>_Agg AggregateBeginningDate : H_STR AggregateBeginningGranuleID : H_STR AggregateBeginningOrbitNumber : H_ILLU AggregateBeginningTime : H_STR AggregateCreationTime : H_STR AggregateCreationDate : H_STR AggregateEndingDate : H_STR AggregateEndingGranuleID : H_STR AggregateEndingOrbitNumber : H_ILLU AggregateEndingTime : H_STR AggregateG-RingLatitude : H_F AggregateG-RIngLongitude : H_F Dataset Array : H_RO Arrays are different for each GEO product. See individual GEO tables for details . <<H_G> > <EDR Collection Short Name> Band_Measurement_Mode_ID : H_STR N_Anc_Type_Tasked : N_STR N_Collection_Short_Name : H_STR N_Dataset_Type_Tag : H_STR N_Instrument_Flight_SW_Version : H_I N_Processing_Domain : H_STR N_Spacecraft_SW_Version : H_STR Operational_Mode : H_STR Processing_Level_Identifier : H_STR N_Num_Granules : H_I 1 1 <<H_G>> <GEO Collection Short Name> 1.. n <<H_DS>> <GEO Collection Short Name>_Gran_n One or more EDR_Product Groups may be present for each Data_Products Group <<Root> >/ (EDR) Identifier_Information.Dataset_Indentifier : H_STR Instrument_Short_Name : H_STR Mission_Name : H_STR N_Creation_Date : H_STR N_Creation_Time : H_STR N_Dataset_Source : H_STR N_Document_Ref : H_STR N_GEO_Ref : H_STR N_Subsystem_SW_Version : H_STR N_Update_Date : H_STR N_Update_Time : H_STR Platform_Short_Name : H_STR <<H_DS> > <EDR Collection Short Name>_Gran_n Ascending/Decending_Indicator : H_C Beginning_Date : H_STR Beginning_Time : H_STR Cloud_Cover : H_F East_Bounding_Coordinate : H_F Ending_Date : H_STR Ending_Time : H_STR G-Ring_Latitude : H_F Array G-Ring_Longitude : H_F Array Input_Dataset_Identifier.Dataset_Identifier : H_STR N_Algorithm_Version : H_STR N_Anc_Filename : H_STR N_Aux_Filename : H_STR N_Beginning_Orbit_Number : H_ILLU N_Beginning_Time_IET : H_ILL N_Day_Night_Flag : H_STR N_Degraded_Mode : H_C N_Ending_Time_IET : H_ILL N_Granule_ID : H_ILL N_Granule_Quality_Value : H_F N_Granule_Quality_Value_Explanation : H_STR N_Granule_Version : H_STR N_LEO_Flag : H_STR N_LUT_Version : H_STR N_Nadir_Latitude_Max : H_F N_Nadir_Latitude_Min : H_F N_Nadir_Longitude_Max : H_F N_Nadir_Longitude_Min : H_F N_Percent_Erroneous_Data : H_F N_Percent_Missing_Data : H_F N_Percent_Not-Applicable_Data : H_F N_Reference_ID : H_STR N_Satellite/Local_Azimuth_Angle_Max : H_F N_Satellite/Local_Azimuth_Angle_Min : H_F N_Satellite/Local_Zenith_Angle_Max : H_F N_Satellite/Local_Zenith_Angle_Min : H_F N_Solar_Azimuth_Angle_Max : H_F N_Solar_Azimuth_Angle_Min : H_F N_Solar_Zenith_Angle_Max : H_F N_Solar_Zenith_Angle_Min : H_F N_Spacecraft_Maneuver : H_STR North_Bounding_Coordinate : H_F South_Bounding_Cooridinate : H_F West_Bounding_Coordinate : H_F Dataset Array : H_RR 1.. n 1 <<H_DS> > <EDR Collection Short Name>_Agg AggregateBeginningDate : H_STR AggregateBeginningGranuleID : H_STR AggregateBeginningOrbitNumber : H_ILLU AggregateBeginningTime : H_STR AggregateCreationDate : H_STR AggregateCreationTime : H_STR AggregateEndingDate : H_STR AggregateEndingGranuleID : H_STR AggregateEndingOrbitNumber : H_ILLU AggregateEndingTime : H_STR AggregateG-RingLatitude : H_F Array AggregateG-RingLongitude : H_F Array AggregatePercentMissingDataAve : H_F Dataset Array : H_RO Arrays are different for each EDR product. See individual EDR tables for details . 24

Ancillary/Auxiliary UML Models <<Root>> AUX_HDF_File_Root group <<Root>> ANC_HDF_File_Root Group Ancillary_Dataset_Identifier.Dataset_Identifier : H_STR Ancillary_Dataset_Identifier.Dataset_Identifier : H_STR Identifier_Information.Dataset_Indentifier : H_STR Input_Dataset_Identifier.Dataset_Identifier : H_STR Instrument_Short_Name : H_STR Mission_Name : H_STR N_Anc_Type : H_STR N_Beginning_Time_IET : H_LL N_Creation_Date : H_STR N_Creation_Time : H_STR N_Dataset_Source : H_STR N_Dataset_Type_Tag : H_STR N_Ending_Time_IET : H_LL N_LEO_Flag : H_STR N_Processing_Domain : H_STR N_Spacecraft_SW_Version : H_STR N_Subsystem_SW_Version : H_STR N_Update_Date : H_STR N_Update_Time : H_STR Platform_Short_Name : H_STR Identifier_Information.Dataset_Indentifier : H_STR Input_Dataset_Identifier.Dataset_Identifier : H_STR Mission_Name : H_STR N_Anc_Type : H_STR N_Creation_Date : H_STR N_Creation_Time : H_STR N_Dataset_Source : H_STR N_Dataset_Type_Tag : H_STR N_Processing_Domain : H_STR N_Update_Date : H_STR N_Update_Time : H_STR 1 1 1 1 <<H_G>> HDF_All_Data 1 1..* <<H_G>> HDF_xxx_Product_Data 1 <<H_G>> AncillaryData 1 <<H_G>> HDF_All_Data 1 <<H_G>> AuxiliaryData (from ANC_HDF_File_Root Group) Calander_Date : H_STR Time_of_Day : H_STR (from AUX_HDF_File_Root group) 1 1..* Calendar_Date N_Granule_Version : H_STR Time_of_Day : H_STR <<H_G>> HDF_xxx_Product_Data 1 1..* <<H_DS>> xxx_Data 1..* <<H_DS>> xxx_Data 25

NPOESS Sample Data Patrick Purcell

Product Example UML Model <<Root> > / (EDR) <<H_G> > Data_Product s <<H_G> > <EDR Collection Short Name> Band_Measurement_Mode_ID : H_STR N_Anc_Type_Tasked : N_STR N_Collection_Short_Name : H_STR N_Dataset_Type_Tag : H_STR N_Instrument_Flight_S W_Version : H_I N_Processing_Domain : H_STR N_Spacecraft_SW_Version : H_STR Operational_Mode : H_STR Processing_Level_Identifier : H_STR N_Num_Granules : H_I <<H_DS> > <EDR Collection Short Name>_Agg AggregateBeginningDate : H_STR AggregateBeginningGranuleID : H_STR AggregateBeginningOrbitNumber : H_ILLU AggregateBeginningTime : H_STR AggregateCreationDate : H_STR AggregateCreationTime : H_STR AggregateEndingDate : H_STR AggregateEndingGranuleID : H_STR AggregateEndingOrbitNumber : H_ILLU AggregateEndingTime : H_STR AggregateG-RingLatitude : H_F Array AggregateG-RingLongitude : H_F Array AggregatePercentMissingDataAve : H_F Dataset Array : H_RO <<H_DS> > <EDR Collection Short Name>_Gran_n Ascending/Decending_Indicator : H_C Beginning_Date : H_STR Beginning_Time : H_STR Cloud_Cover : H_F East_Bounding_Coordinate : H_F Ending_Date : H_STR Ending_Time : H_STR G-Ring_Latitude : H_F Array G-Ring_Longitude : H_F Array Input_Dataset_Identifier.Dataset_Identifier : H_STR N_Algorithm_Version : H_STR N_Anc_Filename : H_STR N_Aux_Filename : H_STR N_Beginning_Orbit_Number : H_ILLU N_Beginning_Time_IET : H_ILL N_Day_Night_Flag : H_STR N_Degraded_Mode : H_C N_Ending_Time_IET : H_ILL N_Granule_ID : H_ILL N_Granule_Quality_Value : H_F N_Granule_Quality_Value_Explanation : H_STR N_Granule_Version : H_STR N_LEO_Flag : H_STR N_LUT_Version : H_STR N_Nadir_Latitude_Max : H_F N_Nadir_Latitude_M : in H_F N_Nadir_Longitude_Max : H_F N_Nadir_Longitude_Min : H_F N_Percent_Erroneous_Data : H_F N_Percent_Missing_Data : H_F N_Percent_Not-Applicable_Data : H_F N_Reference_ID : H_STR N_Satellite/Local_Azimuth_Angle_Max : H_F N_Satellite/Local_Azimuth_Angle_M : in H_F N_Satellite/Local_Zenith_Angle_Max : H_F N_Satellite/Local_Zenith_Angle_Min : H_F N_Solar_Azimuth_Angle_Max : H_F N_Solar_Azimuth_Angle_Min : H_F N_Solar_Zenith_Angle_Max : H_F N_Solar_Zenith_Angle_Min : H_F N_Spacecraft_Maneuver : H_STR North_Bounding_Coordinate : H_F South_Bounding_Cooridinate : H_F West_Bounding_Coordinate : H_F Dataset Array : H_RR <<H_RO Array>> (EDR VIIRS I1-Channel Imagery) ( from <EDR Collection Short Name>_Agg) radianceI1_Array : H_RO reflectanceI1_Array : H_RO radianceI1_Factors : H_RO reflectanceI1_Factors : H_RO <<H_RR Array>> (EDR VIIRS I1-Channel Imagery) ( from <EDR Collection Short Name>_Gran_n) radianceI1_Array : H_RR reflectanceI1_Array : H_RR radianceI1_Factors : H_RR reflectanceI1_Factors : H_RR These arrays are different for each product. 27

HDF5 View - ImageView 28

Product Profile Example Name Radiance I1 Array Fields Data Field Dimensions Size Offset 2bytes 0 Name Attribute Name Granule Boundary Location Dynamic Field Names Max Index Min Index inTrack inTrack Yes Product Yes crossTrack crossTrack No No 8241 8241 Datum Name Offset Scale Measurement Range Range Data Fill Values Units Min Max Type Radiance 0 1800 Int16 Name Dynamic Attribute Data Location W/(m2⋅sr⋅µm) 0 I1 Pixel Name Type NA_INT16_FILL Yes radianceI1 DataSet MISS_INT16_FILL Factors ONBOARD_PT_INT16_FILL ONGROUND_PT_INT16_FILL ERR_INT16_FILL Reflectanc 2bytes 0 e I1 Array Value -999 -998 -997 -996 -995 Legend Entries Name Value Name Attribute Name Granule Boundary Location Dynamic Field Names Max Index Min Index inTrack inTrack Yes Product Yes crossTrack crossTrack No No 8241 8241 Datum Name Offset Scale Reflectanc 0 e I1 Pixel Measuremen Range Range Data Fill Values t Units Min Max Type 0 1 INT16 Name Dynamic Attribute Data Locatio Unitless Name Type n NA_INT16_FILL Yes reflectanceI DataSet 1 Factors MISS_INT16_FILL Legend Entries Value Nam Value e -999 -998 ONBOARD_PT_INT16_FILL -997 ONGROUND_PT_INT16_FILL -996 ERR_INT16_FILL -995 29

HDF5 API – C++ Reader Sample Code Note: A granule holds a reference to a dataset region ... ... while an aggregation contains a reference to an object. 30

NPOESS HDF5 Summary • • • • • • The NPOESS Program delivers the official deliverable data products (RDR, SDR/TDR, EDR/ARP/IP) and dynamic ancillary data and auxiliary data in HDF5 Files The HDF5 Files have an XML User Block that can be accessed without HDF5 tools - provides a “quick-look” into the metadata before opening the HDF5 file Metadata within the HDF5 files are stored as attributes There are general UML Models for the NPOESS official delivered data that provide a common framework Official deliverable data products are organized by reference objects (aggregations) which contain one or more reference regions (granules) Although data may be accessed directly through the All_Data group, the Data_Products group provides integrated access: – Allows the user to access both metadata and data through a common HDF5 group • Metadata is accessed directly by reading the Attribute values • Datasets may be accessed by dereferencing the object ID stored in the Data_Products Group for the aggregation or granule 31

Add a comment

Related presentations

Related pages

Profile of NPOESS HDF5 Files - Technology

The NPOESS program uses Unified Modeling Language (UML) to describe the format of the HDF5 files produced. For each unique type of data product, the HDF5 ...
Read more

PowerPoint Presentation - HDFEOS.org

Profile of National Polar-Orbiting Operational Satellite System (NPOESS) HDF5 Files Kim Tomashosky, Ken Stone, Pat Purcell, Ron Andrews NPOESS Program
Read more

NPP/ NPOESS Product Data Format - HDFEOS.org

... are in separate product profile file. ... so it is important to standardize how NPOESS product data is organized in HDF5. Enhancements by “ned ...
Read more

Hdf5 | LinkedIn

Download the following two hdf5 files: ... Profile of NPOESS HDF5 Files. 120 Views. HDFEOS. Using HDF5 and Python: The H5py module. 5,174 Views. HDFEOS.
Read more

A Proposal to Provide HDF Support for

... 2007 HDF Support for the ESDIS ... We submitted a new version of the “HDF5 Data Model, File Format and ... We provided assistance to NPOESS ...
Read more

HDF5 NPP Data Format

HDF5 NPP Data Format ... General NPP HDF5 File Structure / AllData Data_Products _All
Read more

PPT - National Polar-orbiting Operational Environmental ...

Profile of National Polar-Orbiting Operational Satellite System NPOESS HDF5 Files. National Polar-orbiting Operational Environmental ... (NPOESS) HDF5 Files.
Read more

HDF5 NPP Data Format - STAR - NOAA / NESDIS / Center for ...

HDF5 NPP Data Format ... •Quality flags are stored in the HDF5 files as N number(s) of two or three dimensional, ... profile. •Dimensions are ...
Read more

Presentation "HDF5 for NPOESS Data Products Alan M ...

... where is defined by FGDC or the NPOESS extension profile. l Metadata will be hierarchical, ... (NPOESS) HDF5 Files Kim Tomashosky, Ken Stone, ...
Read more