Skip to content

Log VOL - an HDF5 VOL connector for storing data in a time-log layout in files

Notifications You must be signed in to change notification settings

HDFGroup/vol-log-based

Repository files navigation

Log VOL - an HDF5 VOL connector for storing data in a time-log layout in files

This software repository contains source codes implementing Log VOL connector, an HDF5 Virtual Object Layer (VOL) plugin that stores HDF5 datasets in a storage layout similar to the time log. When using the Log VOL connector, write requests from an MPI process are appended one after another (as logs) in a contiguous space in the file. The contiguous spaces of multiple processes are appended one after another following the increasing order of processes' MPI rank IDs. Such a log layout I/O strategy avoids the expensive inter-process communication required when storing data in the canonical order. One of the pwoerful features of HDF5 VOL is to allow applications to make use a VOL connector by setting two environment variables without changing the application source codes. Using the Log VOL connector, exist HDF5 programs can achieve a better parallel write performance with no changes to their codes. Files created by the Log VOL conform with the HDF5 file format specification, but require the Log VOL to read them back.

  • Current build status:
    • Ubuntu_mpich
    • Ubuntu with OpenMPI
    • Mac with MPICH

HDF5 VOL Connector ID

Documents

Application Case Studies and Experimental Results

Developers

Copyright (C) 2022, Northwestern University. See COPYRIGHT notice in top-level directory.

Project funding supports:

Ongoing development and maintenance of the Log VOL connector are supported by the Exascale Computing Project (17-SC-20-SC), a joint project of the U.S. Department of Energy's Office of Science and National Nuclear Security Administration, responsible for delivering a capable exascale ecosystem, including software, applications, and hardware technology, to support the nation's exascale computing imperative.