HDF5 Software Process

33 %
67 %
Information about HDF5 Software Process
Technology

Published on February 18, 2014

Author: HDFEOS

Source: slideshare.net

Description

This talk presents the HDF Group's approach to software engineering. We will share with our users day-to-day maintenance practices and operations at THG along with the future steps to take to assure robustness, sustainability and low cost maintenance of the HDF software.

HDF5 Software Process MuQun Yang, Quincey Koziol, Elena Pourmal The HDF Group 11/7/2007 HDF and HDF-EOS Workshop XI, Landover, MD 1

Purposes • Demonstrate how we maintain HDF5 - Libraries and tools built on top of HDF5 • HDF-EOS5, NetCDF4 and Pytables etc • Hear your feedback 11/7/2007 HDF and HDF-EOS Workshop XI, Landover, MD 2

Three pillars for robust software • Correctness • Performance • Coding standard 11/7/2007 HDF and HDF-EOS Workshop XI, Landover, MD 3

HDF5 software challenges - Portability • Portability: IBM,SGI,windows, linux, Solaris, OSF1, cygwin, Cray,FreeBSD, Mac-OS • Parallel IO: depends on MPI-IO, parallel File System and hardware - MPI-IO: IBM AIX, MPICH, SGI Altix - Parallel File System: GPFS, Lustre 11/7/2007 HDF and HDF-EOS Workshop XI, Landover, MD 4

HDF5 software challenges - Features • Programming languages - C, Fortran, C++ • External libraries: szlib encoder and decoder, zlib • Comprehensive internal library test suite - time-consuming tests: fractral heap 11/7/2007 HDF and HDF-EOS Workshop XI, Landover, MD 5

HDF5 software challenges - Others • 34 configuration features --enable-cxx, --enable-fortran etc. • THE TESTING CHALLENGE machines x operating systems x compilers x languages x Szip (encoder + no encoder) x (serial + parallel) = a very large number • Coordination among developers - 3-4 core library developers 5-6 developers for tools and others - subversion not enough 11/7/2007 HDF and HDF-EOS Workshop XI, Landover, MD 6

Solutions • HDF5 Daily Test on main-stream UNIX platforms - Rob Matzke started around 1997 - Albert Cheng took over • • • • More platforms, testing with more features Different version of HDF5 1.6, 1.8 Other product: HDF4 Other platforms: Windows 11/7/2007 HDF and HDF-EOS Workshop XI, Landover, MD 7

Daily automatic test procedure art the automatic job Configuring Compiling library and tools Running tests for library and tools Installing the library Testing examples The developer fixes the problem 11/7/2007 Sending out the results to hdf5 library mailing lists 1. Platform watcher diagnoses the failure 2. Inform the corresponding developer if the failure is real HDF and HDF-EOS Workshop XI, Landover, MD 8

An example for daily test Date:  Tue, 6 Nov 2007 08:00:15 -0600 [08:00:15 AM CST] From:  HDF Tester hdftest@hdfgroup.org To:  hdf5lib@hdfgroup.org Subject:  kagiso HDF5_Daily_Tests_1106Tue_FAILED!!! Date:  Tue, 6 Nov 2007 09:23:49 -0600 [09:23:49 AM CST] From:  Quincey Koziol <koziol@hdfgroup.org> To:  hdf5lib@hdfgroup.org Subject: Re: kagiso HDF5_Daily_Tests_1106Tue_FAILED!!!   11/7/2007 HDF and HDF-EOS Workshop XI, Landover, MD 9

11/7/2007 HDF and HDF-EOS Workshop XI, Landover, MD 10

Other helpers • Committest script -automatically test a few platforms before checking in source code • Save developers’ time 11/7/2007 HDF and HDF-EOS Workshop XI, Landover, MD 11

Performance • High IO performance is always a goal for THG • Detect bad performance in time • Performance framework 11/7/2007 HDF and HDF-EOS Workshop XI, Landover, MD 12

Performance framework • Easy to Use for Various Benchmarks • Multiple Platforms and Versions • Long Term Regression Tests • Help Debugging 11/7/2007 HDF and HDF-EOS Workshop XI, Landover, MD 13

Background • Backend: Cron job / DB Storage • Core: Performance C/C++ Library • Frontend: PHP / jpgraph 11/7/2007 HDF and HDF-EOS Workshop XI, Landover, MD 14

Solution Easy to Use HDF5 1.6 HDF5 1.8 cron A User’s Benchmark Database Performance Library www PHP Web Server Graph/Text 11/7/2007 HDF and HDF-EOS Workshop XI, Landover, MD 15

Example Usage H5Perf_startTimer(&time); for(i=0;i<1000 ;i++) { H5Gcreate(fileid,group_name,(size_t)0)); // Add groups } H5Perf_endTimer(&time); H5Perf_addInstance(db_host, date, time); 00 21 * * * /home/local/hyoklee/src/chicago/test-perf-hdfdap-3.sh | 178820 | 2007-08-17 21:51:14 | 10000 groups Timestamp 11/7/2007 Instance Name | creating 10000 empty groups | 1.8.0 | hdfdap | 0.670198 | Version Platform HDF and HDF-EOS Workshop XI, Landover, MD Time 16 4384 |

Demo http://hdfdap.hdfgroup.uiuc.edu/h5perf/index.html 11/7/2007 HDF and HDF-EOS Workshop XI, Landover, MD 17

Other Performance work • Performance studies compression, chunking and parallel IO http://www.hdfgroup.uiuc.edu/papers/papers/ 11/7/2007 HDF and HDF-EOS Workshop XI, Landover, MD 18

Coding standard • Not much except seminars on HDF4/HDF5 coding standards – We definitely need to improve in this area 11/7/2007 HDF and HDF-EOS Workshop XI, Landover, MD 19

Other work we have done to improve software process 11/7/2007 HDF and HDF-EOS Workshop XI, Landover, MD 20

User involvements • Public mailing lists hdf-forum@hdfgroup.org hdfnews@hdfgroup.org hdf4dev@hdfgroup.org hdf5dev@hdfgroup.org • Public RFCs • Solicit comments for new HDF5 features etc. http://www.hdfgroup.uiuc.edu/RFC/HDF5/ • Ask special groups to give us feedback http://www.hdfgroup.uiuc.edu/RFC/HDF5/H5CHK/ • Subversion repo 11/7/2007 HDF and HDF-EOS Workshop XI, Landover, MD 21

Trainings for developers • Internally Book reading: Programming Pearls • Attending Dr. Dobb’s software conference 11/7/2007 HDF and HDF-EOS Workshop XI, Landover, MD 22

Near-term plan • Enhance daily correctness tests  API compatibility tests: done  API Version tests: in the process  Java wrapper tests: done  Open source packages that use HDF o EOS2 with HDF4 o EOS5 with HDF5 o NetCDF4 with HDF5 • Weekly “stable” code snapshots 11/7/2007 HDF and HDF-EOS Workshop XI, Landover, MD 23

Long-term Plan • Coding standard: code review • Standards : - In the process of applying for ISO/ANSI standard for HDF5 • 500 random API tests to avoid ungraceful crash • Collect existing HDF5 files such as EOS2, EOS5 files - Running all HDF4/HDF5 tools on these files periodically • Daily correlation regression tests on external machines 11/7/2007 HDF and HDF-EOS Workshop XI, Landover, MD 24

Acknowledgement This work was supported basing upon the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A.  Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA.  11/7/2007 HDF and HDF-EOS Workshop XI, Landover, MD 25

Add a comment

Related presentations

Related pages

Overview of Parallel HDF5 Design - The HDF Group ...

Overview of Parallel HDF5 Design There were several requirements that we had for Parallel HDF5 (PHDF5). These were: Parallel HDF5 files had to be ...
Read more

The HDF Group - Information, Support, and Software

Writing to a Dataset by Chunk In this example each process writes a "chunk" of data to a dataset. The C and Fortran 90 examples result in the same data ...
Read more

NetCDF - Wikipedia, the free encyclopedia

They are also the chief source of netCDF software, standards development, ... NetCDF users can create HDF5 files with benefits not available with the ...
Read more

Hierarchical Data Format - Wikipedia, the free encyclopedia

Hierarchical Data Format; Filename extension ... After a two year review process, ... Huygens Software uses HDF5 as primary storage format since version 3 ...
Read more

HDF Visualization Software - Ab Initio Physics Research

Software for Visualizing HDF Files. Since the MIT Photonic-Bands package outputs its fields and dielectric functions in HDF5 format, it seemed helpful to ...
Read more

hdf-forum - HDF View file process

HDF View file process. Dear all, I have downloaded and installed HDF View in my desktop and downloaded MODIS Sea Surface Temperature images. I am unable to ...
Read more

python - How is HDF5 different from a folder with files ...

How is HDF5 different from a folder with files? ... what software wrote them, ... How can I mmap HDF5 data into multiple Python processes?
Read more

HPC@LSU | Documentation | Software | hdf5

Usage. HDF5 is provided as an application programming interface (API) library, so it must be included during an application build process.
Read more