Automatic reproducibility of COMPSs experiments through the integration of RO-Crate in Chameleon

Mon, 19 Feb 2024 00:00:00 +0000

Topics: Provenance, reproducibility, standards, image creation
Skills: Python, JSON, Bash scripting, Linux, image creation and deployment
Difficulty: Medium
Size: Large (350 hours)
Mentors: Raül Sirvent

Project Idea Description

The COMPSs programming model provides an interface for the programming of a sequential application that is transformed in a workflow that, thanks to the COMPSs runtime, is later scheduled in the available computing resources. Programming is enabled for different languages through the use of bindings: Java, C/C++ and Python (named PyCOMPSs). COMPSs is able to generate Workflow Provenance information after the execution of an experiment. The generated artifact (code + data + recorded metadata) enables the sharing of results through the use of tools such as the WorkflowHub portal, that provides the capacity of generating a DOI of the results to include them as permanent references in scientific papers.

The format of the metadata generated in COMPSs experiments follows the RO-Crate specification, and, more specifically, two profiles: the Workflow and Workflow Run Crate profiles. This metadata enables not only the sharing of results, but also their reproducibility.

This project proposes the creation of a service that enables the automatic reproducibility of COMPSs experiments in the Chameleon infrastructure. The service will be able to get a COMPSs crate (artifact that follows the RO-Crate specification), and, by parsing the available metadata, build a Chameleon compatible image for reproducing the experiment in the testbed. Small modifications to the COMPSs RO-Crate are foreseen (i.e. the inclusion of third party software required by the application).

Project Deliverables

Study the different environments and specifications (COMPSs, RO-Crate, Chameleon, Trovi, …).
Design the most appropriate integration, considering all the elements involved.
Integrate PyCOMPSs basic experiments reproducibility in Chameleon.
Integrate PyCOMPSs complex experiments reproducibility in Chameleon (i.e. with third party software dependencies).

RO-Crate | UCSC OSPO

Automatic reproducibility of COMPSs experiments through the integration of RO-Crate in Chameleon