New scalable I/O techniques for hybrid HPC and Data Intensive workloads (SCIOT

Expand Parallel File System release

Documentation of Expand

Documentation available at:
https://github.com/xpn-arcos/xpn/#readme

The source code of Expand

Source code is at the GitHub repo:
https://github.com/xpn-arcos/xpn/

The Expand Parallel File System

Expand is available at:
https://github.com/xpn-arcos/xpn/releases

Objectives

As said before, the main goal of this project is to create an active I/O stack that dynamically adjusts computation and storage requirements through the use of ad-hoc file systems, new scheduling of storage resources integrated with monitoring along all levels of hybrid storage hierarchies in HEC, and I/O replication and storage malleability techniques to cope with runtime changes of data-intensive applications. TODO:sub-goals:

O1. Enabling the efficient use of new Ad-Hoc file systems to increase application throughput of HPC and cloud systems by leveraging the performance advantage of fast, node-local storage technologies.
O2. Providing data-aware HPC scheduling decisions, based on global monitoring, to reduce I/O interference via globally coordinated minimization of data transfers between storage tiers and in-transit/in-situ processing facilities.
O3. Balancing computation and data transfers by providing elastic mechanisms to dynamically adjust the ratio between the allocations of compute and storage resources.
O4. Creating hybrid storage systems suitable for HPC and Cloud technologies.
O5. Evaluating the feasibility of the proposed solutions through relevant use cases.
O6. To plan and coordinate project activities and to promote results dissemination to increase impact.

Work packages

WP1. Scalable I/O techniques for Ad-Hoc file systems

O1. Enabling the efficient use of new Ad-Hoc file systems. to increase application throughput of HPC and cloud systems by leveraging the performance advantage of fast, node-local storage technologies.

WP2. Monitoring and Data Scheduling

O2. Providing data-aware HPC scheduling decisions, based on global monitoring, to reduce I/O interference via globally coordinated minimisation of data transfers between storage tiers and in-transit/in-situ processing facilities.

WP3. Malleability and replication techniques

O3. Obtain techniques for balancing computation and data transfers by providing elastic mechanisms to dynamically adjust the ratio between the allocations of compute and storage resources.

WP4. New integrated techniques for hybrid data-driven applications in HPC and cloud

O4. Creating hybrid storage systems suitable for HPC and Cloud technologies.

WP5. Integration, validation, and optimization

O5. Evaluating the feasibility of the proposed solutions through relevant use cases.

WP6. Coordination and scheduling of activities

O6. To plan and coordinate project activities and to promote results dissemination to increase impact.

Deliverables

WP4. New integrated techniques for hybrid data-driven applications in HPC and cloud

D4.1. Report on new techniques for I/O interoperability between HPC and cloud systems

WP5. Integration, validation, and optimization

D5.1. Report on use cases and benchmarks

D5.2. Report on Final Evaluation

WP6. Coordination and scheduling of activities

D6.1. Overall project plan and Web page

D6.2. Intermediate Project Report

D6.3. Final report of the project and conclusions

Publications

2025

Improving I/O performance in HPC environments using the Expand Ad-Hoc file system ,
Authors: Diego Camarmas-Alonso, Felix Garcia-Carballeira, Alejandro Calderón, Jesus Carretero,
Journal: The Journal of Supercomputing