Raül Sirvent | UCSC OSPO

Final blog: Automatic reproducibility of COMPSs experiments through the integration of RO-Crate in Chameleon

Thu, 22 Aug 2024 00:00:00 +0000

Introduction

Hello everyone,

I’m Archit from India, an undergraduate student at the Indian Institute of Technology, Banaras Hindu University (IIT BHU), Varanasi. As part of the Automatic Reproducibility of COMPSs Experiments through the Integration of RO-Crate in Chameleon project, my proposal, under the mentorship of Raül Sirvent, aims to develop a service that facilitates the automated replication of COMPSs experiments within the Chameleon infrastructure.

About the Project

The project proposes to create a service that can take a COMPSs crate (an artifact adhering to the RO-Crate specification) and, through analysis of the provided metadata, construct a Chameleon-compatible image for replicating the experiment on the testbed.

Final Product

The basic workflow of the COMPSs Reproducibility Service can be explained as follows:

The service takes the workflow path or link as the first argument from the user.
The program shifts the execution to a separate sub-directory, reproducibility_service_{timestamp}, to store the results from the reproducibility process.
Two main flags are required:
- Provenance flag: If you want to generate the provenance of the workflow via the runcompss runtime.
- New Dataset flag: If you want to reproduce the experiment with a new dataset instead of the one originally used.
If there are any remote datasets, they are fetched into the sub-directory.
The main work begins with parsing the metadata from ro-crate-metadata.json and verifying the files present inside the dataset, as well as any files downloaded as remote datasets. This step generates a status table for the user to check if any files are missing or have modified sizes.

The final step is to transform the compss-command-line.txt and all the paths specified inside it to match the local environment where the experiment will be reproduced. This includes:
- Mapping the paths from the old machine to new paths inside the RO-Crate.
- Changing the runtime to runcompss or enqueue_compss, depending on whether the environment is a SLURM cluster.
- Detecting if the paths specified in the command line are for results, and redirecting them to new results inside the reproducibility_service_{timestamp}\Results directory.
After this, the service prompts the user to add any additional flags to the final command. Upon final verification, the command is executed via Python’s subprocess pipe.

Logging System: All logs related to the Reproducibility Service are stored inside the reproducibility_service_{timestamp}\log.

You can view the basic pseudocode of the service.

Conclusion and Future Work

It’s been a long journey since I started this project, and now it’s finally coming to an end. I have learned a lot from this experience, from weekly meetings with my mentor to working towards long-term goals—it has all been thrilling. I would like to thank the OSRE community and my mentor for providing me with this learning opportunity.

This is only version 1.0.0 of the Reproducibility Service. If I have time from my coursework, I would like to fix any bugs or improve the service further to meet user needs.

However, the following issues still exist with the service and can be improved upon:

Third-party software dependencies: Automatic detection and loading of these dependencies on a SLURM cluster are not yet implemented. Currently, these must be handled manually by the user.
Support for workflows with data_persistence = False: There is no support for workflows where all datasets are remote files.

Deliverables

Reproducibility Service Repository: This repository contains the main service along with guidelines on how to use it. The service will be integrated with the COMPSs official distribution in its next release.
Chameleon Appliance : This is a single-node appliance with COMPSs 3.3.1 installed, so that anyone with access to Chameleon can reproduce experiments.

Previous Blogs

Make sure to check out my other blogs to see how I started this project and the challenges I faced along the way:

Thank you for reading the blog, have a nice day!!

Mid-term Blog: Automatic reproducibility of COMPSs experiments through the integration of RO-Crate in Chameleon

Mon, 29 Jul 2024 00:00:00 +0000

Introduction

Hello everyone I’am Archit from India. An undergraduate student at the Indian Institute of Technology, Banaras Hindu University, IIT (BHU), Varanasi. As part of the Automatic reproducibility of COMPSs experiments through the integration of RO-Crate in Chameleon my proposal under mentorship of Raül Sirvent aims to develop a service that facilitates the automated replication of COMPSs experiments within the Chameleon infrastructure.

About the project:

The project proposes to create a service that will have the capability to take a COMPSs crate (an artifact adhering to the RO-Crate specification) and, through analysis of the provided metadata construct a Chameleon-compatible image for replicating the experiment on the testbed.

Progress

It has been more than six weeks since the ReproducibilityService project began, and significant progress has been made. You can test the actual service from my GitHub repository: ReproducibilityService. Let’s break down what the ReproducibilityService is capable of doing now:

Support for Reproducing Basic COMPSs Experiments: The RS program is now fully capable of reproducing basic COMPSs experiments with no third-party dependencies on any device with the COMPSs Runtime installed. Here’s how it works:
- Getting the Crate: The RS program can accept the COMPSs workflow from the user either as a path to the crate or as a link from WorkflowHub. In either case, it creates a sub-directory for further execution named reproducibility_service_{timestamp} and stores the workflow as reproducibility_service_{timestamp}/Workflow.
- Address Mapping: The ro-crate contains compss_submission_command_line.txt, which is the command originally used to execute the experiment. This command may include many paths such as runcompss flag1 flag2 ... flagn <main_workflow_file.py> input1 input2 ... inputn output. The RS program maps all the paths for <main_workflow_file.py> input1 input2 ... inputn output to paths inside the machine where we want to reproduce the experiment. The flags are dropped as they may be device-specific, and the service asks the user for any new flags they want to add to the COMPSs runtime.
- Verifying Files: Before reproducing an experiment, it’s crucial to check whether the inputs or outputs have been tampered with. The RS program cross-verifies the contentSize from the ro-crate-metadata.json and generates warnings in case of any abnormalities.
- Error Logging: In case of any problems during execution, the std_out and std_err are stored inside reproducibility_service_{timestamp}/log.
- Results: If any results do get generated by the experiment, the RS program stores them inside reproducibility_service_{timestamp}/Results. If we ask for the provenance of the workflow also, the ro-crate thus generated is also stored here only.

Support for Reproducing Remote Datasets: If a remote dataset is specified inside the metadata file, the RS program fetches the dataset from the specified link using wget, stores the remote dataset inside the crate, and updates the path in the new command line it generates.

Challenges and End-Term Goals

Support for DATA_PERSISTENCE_FALSE: The RS program still needs to support crates with dataPersistence set to false. After weeks of brainstorming ideas on how to implement this, we recently concluded that since the majority of DATA_PERSISTENCE_FALSE crates are run on SLURM clusters, and the dataset required to fetch in such a case is somewhere inside the cluster, the RS program will support this case for such clusters. Currently, I am working with the Nord3v2 cluster to further enhance the functionality of ReproducibilityService.
Chameleon Cluster Setup: I have made some progress towards creating a new COMPSs 3.3 Appliance on Chameleon to test the service. However, creating the cluster setup script needed for the service to run on a COMPSs 3.3.1 cluster to execute large experiments has been challenging.
Integrating with COMPSs Repository: After completing the support for dataPersistence false cases, we aim to launch this service as a tool inside the COMPSs repository. This will be a significant milestone in my developer journey as it will be the first real-world project I have worked on, and I hope everything goes smoothly.

Stay tuned for the next blog!!

Automatic reproducibility of COMPSs experiments through the integration of RO-Crate in Chameleon

Wed, 12 Jun 2024 00:00:00 +0000

About the project:

How it all started

This journey began amidst our college’s cultural fest, in which I was participating, just 15 days before the proposal submission deadline. Many of my friends had been working for months to get selected for GSoC. I didn’t think I could participate this year because I was late, so I thought, “Better luck next year.” But during the fest, I kept hearing about UC OSPO and that a senior had been selected within a month. So, I was in my room when my friend told me, “What’s the worst that can happen? Just apply,” and so I did. I chose this project and wrote my introduction in Slack without knowing much. After that, it’s history. I worked really hard for the next 10 days learning about the project, making the proposal, and got selected.

First few weeks:

I started the project a week early from June 24, and it’s been two weeks since. The start was a bit challenging since it required setting up a lot of things on my local machine. For the past few weeks, the majority of my time has been dedicated to learning about COMPSs, RO-Crate, and Chameleon, the three technologies this project revolves around. The interaction with my mentor has also been great. From the weekly report meetings to the daily bombardment of doubts by me, he seems really helpful. It is my first time working with Chameleon or any cloud computing software, so it can be a bit overwhelming sometimes, but it is getting better with practice.

Stay tuned for progress in the next blog!!

Automatic reproducibility of COMPSs experiments through the integration of RO-Crate in Chameleon

Mon, 19 Feb 2024 00:00:00 +0000

Topics: Provenance, reproducibility, standards, image creation
Skills: Python, JSON, Bash scripting, Linux, image creation and deployment
Difficulty: Medium
Size: Large (350 hours)
Mentors: Raül Sirvent

Project Idea Description

The COMPSs programming model provides an interface for the programming of a sequential application that is transformed in a workflow that, thanks to the COMPSs runtime, is later scheduled in the available computing resources. Programming is enabled for different languages through the use of bindings: Java, C/C++ and Python (named PyCOMPSs). COMPSs is able to generate Workflow Provenance information after the execution of an experiment. The generated artifact (code + data + recorded metadata) enables the sharing of results through the use of tools such as the WorkflowHub portal, that provides the capacity of generating a DOI of the results to include them as permanent references in scientific papers.

The format of the metadata generated in COMPSs experiments follows the RO-Crate specification, and, more specifically, two profiles: the Workflow and Workflow Run Crate profiles. This metadata enables not only the sharing of results, but also their reproducibility.

This project proposes the creation of a service that enables the automatic reproducibility of COMPSs experiments in the Chameleon infrastructure. The service will be able to get a COMPSs crate (artifact that follows the RO-Crate specification), and, by parsing the available metadata, build a Chameleon compatible image for reproducing the experiment in the testbed. Small modifications to the COMPSs RO-Crate are foreseen (i.e. the inclusion of third party software required by the application).

Project Deliverables

Study the different environments and specifications (COMPSs, RO-Crate, Chameleon, Trovi, …).
Design the most appropriate integration, considering all the elements involved.
Integrate PyCOMPSs basic experiments reproducibility in Chameleon.
Integrate PyCOMPSs complex experiments reproducibility in Chameleon (i.e. with third party software dependencies).