Implementing HDF5 in MATLAB

60 %
40 %
Information about Implementing HDF5 in MATLAB

Published on February 18, 2014

Author: HDFEOS



The MathWorks introduced MATLAB support for HDF5 in 2002 via three high-level functions: HDF5INFO, HDF5READ, and HDF5WRITE. These functions worked well for their purpose-providing simple interfaces to a complicated file format-but MATLAB users requested finer control over their HDF5 files and the HDF5 library. MATLAB 7.3 (R2006b) adds this precise level of support for version 1.6.5 of the HDF5 library via a close mapping of the HDF5 C API to MATLAB function calls.

This presentation will briefly introduce the earlier, high-level HDF5 interface (and its limitations) before showing in detail the low-level HDF5 functions. It will show how to interact with the HDF5 library and files using the thirteen classes of functions in MATLAB, which encapsulate groupings of functionality found in the HDF5 C API. But because MATLAB is itself a higher-level language than C, we will also present MATLAB's extensions and modifications of the HDF5 C API that make it more MATLAB-like, work with defined values, and perform ID and memory management.

Wrapping a library like HDF5 requires a great deal of effort and design, and we will briefly present a general-purpose mechanism for creating close mappings between library interfaces and an application like MATLAB. One of our goals in this presentation is to facilitate communication with The HDF Group about how The MathWorks builds our HDF5 interfaces in order to ease adoption of future versions of the HDF5 library in large, general-purpose applications.

Jeff Mather & Alec Rogers The MathWorks, Inc. 29 November 2006 © 2006 The MathWorks, Inc. Implementing HDF5 in MATLAB

HDF4 1-1 mapping of C API first. (1998) Customer requests for high-level functions. HDFREAD, HDFWRITE, HDFINFO. (2000) 2

HDF5 High-level first. (2003) Customer requests for lower-level functionality. 1-1 mapping of C API. (2006) 3

The World of HDF Applications HDF4 / HDF5 APIs API Supported by MATLAB High-level access functions Customer application Customer application Customer application Customer application 4

HDF5READ DATA = HDF5READ(FILENAME,DATASETNAME) returns in the variable DATA all data from the file FILENAME for the data set named DATASETNAME. DATA has to be extremely general because of the wide variety of datatypes that HDF5 accomodates. More control needed to match the uniqueness of customer datasets and files. Simple access only: ● No subsetting. ● Limited datatype control. 5

HDF5INFO FILEINFO = HDF5INFO(FILENAME) returns a structure whose fields contain information about the contents of an HDF5 file. FILENAME is a string that specifies the name of the HDF file. 6

HDF5WRITE HDF5WRITE(FILENAME, LOCATION, DATASET) adds the data in DATASET to the HDF5 file named FILENAME. LOCATION defines where to write the DATASET in the file and resembles a Unix-style path. The data in DATASET is mapped to HDF5 datatypes using the rules below. . . . HDF5WRITE is completely symmetric with HDF5READ. Objects disambiguate datatypes. The values in DATASET are cumbersome for non-native MATLAB types (e.g., arrays, compound, and references). 7

Customer HDF5 Requests         Library upgrades (1.4.5, 1.6.4, 1.6.5, 1.8) Better support for large data Hyperselection, chunking New platform support (Solaris 64, MacIntel) GZIP, SZIP compression HDF5 file interrogation Bitfield, date/time datatypes Data translators: HDF5 --> MATLAB 8

Use Cases Read parts of an HDF5 dataset (a hyperslab). 9

Use Cases Read complicated datatypes without the overhead of MATLAB objects for datasets. mydata(1).Data(1).Data(1) 10

Use Cases Allow users to extend our HDF5 functionality without waiting for us. 11

Use Cases Be able to drop in new versions of the HDF5 library when they become available. HDF5 1.8 12

Use Cases Use a variety of esoteric HDF5 features at once: “I'm trying to use HDF5 files [with] grouping features like compound data types, group links, and reference data types.” 13

Schedule Iterative development Internal design reviews March 2006 First draft specification Sept. 2005 Final specification Feb. 2006 Feature complete May 2006 14

MATLAB is not C Hmm . . . I might throw an error. MATLAB [out1, out2] = function(in1, in2); C status = function(in1, in2, &out1, &out2); 15

MATLAB is not C mxArray void * p_real void * p_complex size_t dims[] size_t ndims mxCLASS_ID type ... ... 16

The Interface hid_t identifiers C API MATLAB API H5Xfcn H5X.fcn A D Identifier objects × 12 F G ML S Exceptions T ... 17

Special MATLAB Functions H5ML.compare_values H5ML.get_constant_names H5ML.get_constant_value H5ML.get_function_names H5ML.get_mem_datatype H5ML.hoffset H5ML.sizeof 18

Library Model HDF5 Library Library Constants Library Procedures Lefthand to righthand mapping Procedures Parameters Datatype Conversions 19

Implementing the HDF5 Library       Step 1: Determine auto vs. manual conversion Step 2: Convert .h to .xml Step 3: Convert XML to C++ Step 4: Code manual functions Step 5: Integrate Step 6: Test 20

The conversion process 21

Converting XML to C++ // Definition #define ADD_PROCEDURE_1_5(name,pfn,ret,a1,a2,a3,a4,a5) addMethod(new LibraryProcedure_1_5< LibraryParameter_T<ret>, LibraryParameter_T<a1>, LibraryParameter_T<a2>, LibraryParameter_T<a3>, LibraryParameter_T<a4>, LibraryParameter_T<a5> > (name, atts, pfn)); 22

Converting XML to C++ // Usage (x ~220 functions) atts.init(0,1,5,5); atts.setParamFlags(0, ParameterAttributes::OUTPUT, 1); atts.setParamFlags(1, ParameterAttributes::INPUT | ParameterAttributes::STRING_CONVERT, 1); atts.setParamFlags(2, ParameterAttributes::INPUT, 1); atts.setParamFlags(3, ParameterAttributes::INPUT | ParameterAttributes::STRING_CONVERT, 1); atts.setParamFlags(4, ParameterAttributes::INPUT | ParameterAttributes::STRING_CONVERT, 1); atts.setParamFlags(5, ParameterAttributes::INPUT | ParameterAttributes::STRING_CONVERT, 1); ADD_PROCEDURE_1_5("H5Acreate", H5Acreate, hid_t, hid_t, const char *, hid_t, hid_t, hid_t); 23

The HDF Group and The MathWorks     Continue to communicate future directions. Don't change the existing API functions. Communicate API functionality changes. Produce a machine parsable version of hdf5.h. 24

The Future HDF5 MAT-File 64-bits for large arrays ● Data subsetting on load ● Type conversion on load ● Parallel I/O? ● 25

Add a comment

Related presentations

Presentación que realice en el Evento Nacional de Gobierno Abierto, realizado los ...

In this presentation we will describe our experience developing with a highly dyna...

Presentation to the LITA Forum 7th November 2014 Albuquerque, NM

Un recorrido por los cambios que nos generará el wearabletech en el futuro

Um paralelo entre as novidades & mercado em Wearable Computing e Tecnologias Assis...

Microsoft finally joins the smartwatch and fitness tracker game by introducing the...

Related pages

Implementing HDF5 in MATLAB -

2 0 0 6 T h e M a t h W o r k s, In c. Implementing HDF5 in MATLAB Jeff Mather & Alec Rogers The MathWorks, Inc. 29 November 2006
Read more

Using HDF5 in MATLAB - Technology -

Using HDF5 in MATLAB Jun 11, 2015 Technology the-hdf-eos-tools-and-information-center. The document was removed. ... Implementing HDF5 in MATLAB.
Read more

Software using HDF5: Descriptions - The HDF Group

Software using HDF5: Descriptions. Freely Available Software Commercial Software. Freely Available Software Bathymetric Attributed Grid data format
Read more

Hdf5 - Documents

Documents; Hdf5; Hdf5 Dec 18, 2014 Documents smith-kim. of 29 ... Implementing HDF5 in MATLAB. Overview of Parallel HDF5 and Performance Tuning in HDF5 ...
Read more

Using HDF5 in MATLAB - Documents -

Implementing HDF5 in MATLAB The MathWorks introduced MATLAB support for HDF5 in 2002 via three high-level functions: HDF5INFO, HDF5READ, and HDF5WRITE.
Read more

HDF5 Diskmap Class - File Exchange - MATLAB Central

HDF5 Diskmap Class. ... This is due to implementing direct calls to the HDF5 mex function. ... This file inspired Hdf5 Diskmap Class. MATLAB release:
Read more

How do I change the data type in an HDF5 file from MATLAB ...

How do I change the data type in an HDF5 file from MATLAB? up vote 3 down vote favorite. ... Implementing atoi() in C Invited talk: etiquette ...
Read more

HDF5 (.h5)—Wolfram Language Documentation

Computable Document Format Computation-powered interactive documents. Wolfram Engine Software engine implementing the Wolfram Language. Wolfram Natural ...
Read more