Kate Keahey | UCSC OSPO

Chameleon Trovi Support for Complex Experiment Appliances

Tue, 18 Feb 2025 00:00:00 +0000

Overview

The discoverability and accessibility of research artifacts remains a significant barrier to reproducibility in computer science research. While digital libraries index research papers, they rarely provide direct access to the artifacts needed to reproduce experiments, especially complex multi-node systems. Additionally, when artifacts are available, they often lack standardized metadata, versioning, and deployment mechanisms that would enable researchers to easily find and reuse them. This project addresses these challenges by extending Trovi, a repository of experimental artifacts executable on open platforms, to support complex, multi-node appliances, making sophisticated experimental environments discoverable, shareable, and deployable through a standardized interface - ultimately lowering the barriers to reproducing complex systems experiments.

Chameleon has historically enabled researchers to orchestrate complex appliances—large, multi-node clusters configured via OpenStack Heat—to conduct advanced experiments. Meanwhile, Chameleon team introduced Trovi as repository for open platforms (beyond Chameleon) that pioneers mechanisms for artifact and platform integration leading to immediate execution for pratical reproducibility. This project aims to bridge the two by adding support in Trovi for importing, discovering, and launching complex appliances. By integrating these capabilities, researchers will be able to one-click deploy complex appliances directly from the Trovi dashboard, archive them for future reference, and reproduce experiments on demand.

Key Outcomes

Extended Trovi API: Enable the import and management of complex appliances as artifacts.
Streamlined One-Click Launch: Integrate with Chameleon’s existing provisioning workflows so users can launch multi-node clusters directly from Trovi.
Enhanced Dashboard Experience: Provide UI assistance for discovering, reviewing, and customizing complex appliance artifacts.
Improved Artifact Reproducibility: Automate the process of exporting CC-snapshot images and other resources to ensure everything is preserved across sites (UC, TACC), highlighting any parameters that need user attention for cross-site portability.

Topics: Reproducible Research, Cloud Computing & Orchestration, OpenStack Heat, UI/UX & Web Development

Skills: Python, APIs, Cloud (OpenStack), DevOps & Automation, Frontend

Difficulty: Hard

Size: Large

Mentors: Mark Powers

Tasks:

Extensions to the Trovi API
- Add support for importing complex appliances as artifacts (including Heat templates, metadata, and associated disk images).
- Develop methods for tagging, versioning, and categorizing these appliances, making them easier to discover.
One-Click Launch of Complex Appliances
- Integrate with Chameleon’s orchestration engine, enabling single-click cluster deployments from the Trovi UI.
- Validate correct configuration and resource availability through automated checks.
Trovi Dashboard Enhancements
- Update the front-end to provide intuitive controls for customizing or parameterizing complex appliances before launching.
- Offer a clear workflow for reviewing dependencies, resource requirements, and usage instructions.
Automated Export & Multi-Site Testing
- Streamline the export of snapshots or images into Trovi as part of the appliance import process.
- Optionally re-run the imported appliances at multiple sites (UC, TACC), detecting any unparameterized settings or missing dependencies.

Chameleon Trovi Redesign

Wed, 21 Feb 2024 13:43:55 -0600

Trovi on Chameleon is an open-source service designed to significantly enhance the practical reproducibility of computer science research. By allowing Chameleon users to upload, share, and access packaged experiments and other research artifacts, Trovi aims to streamline the process of replicating and building upon existing studies. This capability is crucial in the scientific community, where the ability to accurately reproduce research results is as fundamental to validating, critiquing, and extending scientific findings as reading papers. The importance of Trovi lies in its potential to serve as a centralized hub that facilitates the exchange of valuable research outputs, promotes transparency, and fosters collaboration among researchers. By improving the ease with which experiments can be replicated and data can be shared, Trovi supports the advancement of knowledge and innovation in the field of computer science, making it an essential tool for researchers seeking to contribute to the development of reproducible and robust scientific research.

This project will focus on the evolution of Trovi. It will aim to enhance Trovi as a tool to advance practical reproducibility in CS research. Students will evaluate the most important use cases and enabling features necessary to enhance Trovi’s functionality and user experience. With these design insights, students will then create a robust interface that allows researchers to integrate experiment code and data easily as packaged artifacts, similar to the user-friendly design of Google Colab, and build off other users’ artifacts to create novel experiments, similar to the design of GitHub. Furthermore, students will create comprehensive documentation with valuable insights into what works well and what requires improvement, creating a dynamic feedback loop to guide the ongoing redesign process. Lastly, students will actively participate in designing webinars, creating and posting video tutorials, and organizing academic events at the University of Chicago to showcase the work on Trovi. This multifaceted project ensures a well-rounded experience and fosters a collaborative learning environment.

Each of the project ideas below focuses on a different aspect of the overall goal to enhance Trovi as a tool for advancing practical reproducibility in CS research. They are designed to offer a comprehensive approach, from technical development to community engagement, ensuring a well-rounded enhancement of the service.

Topics: User Interface Design User Experience Web Development
Skills: HTML/CSS, JavaScript, UX design principles
Difficulty: Moderate to Hard
Size: Medium to Large
Mentors: Mark Powers
Tasks:
- Conduct user research to understand the needs and pain points of current and potential Trovi users.
- Design wireframes and prototypes that incorporate user feedback and aim to simplify the process of uploading, sharing, and reusing research artifacts.
- Implement the frontend redesign using a modern web framework to ensure responsiveness and ease of use.

Packaged Artifacts Integration System

Topics: Cloud Computing Data Management Web APIs
Skills: Python, RESTful APIs, Docker, Git
Difficulty: Hard
Size: Large
Mentors: Mark Powers
Tasks:
- Develop a system that allows users to easily package and upload their experimental code and data to Trovi.
- Create a standardized format or set of guidelines for packaging experiments to ensure consistency and ease of use.
- Implement API endpoints that enable automated uploads, downloads, and integration with other tools like GitHub or Zenodo.
- Test the system with real-world experiments to ensure reliability and ease of integration.

Community Engagement and Educational Materials

Topics: Educational Technology Community Building Content Creation
Skills: Video Editing, Public Speaking, Event Planning
Difficulty: Moderate
Size: Medium
Mentors: Mark Powers
Tasks:
- Design and organize webinars that introduce Trovi and its new features to the research community.
- Create engaging video tutorials that guide users through the process of using Trovi for their research needs.
- Develop comprehensive documentation that covers both basic and advanced use cases, troubleshooting, and tips for effective collaboration using Trovi.
- Organize academic events, such as workshops or hackathons, that encourage the use of Trovi for collaborative research projects.

Feedback Loop and Continuous Improvement System

Topics: Software Engineering Data Analysis User Feedback
Skills: Python, SQL, Data Visualization, Web Development
Difficulty: Moderate
Size: Medium
Mentors: Mark Powers
Tasks:
- Implement a system within Trovi for collecting, storing, and analyzing user feedback and usage data.
- Develop dashboards that visualize feedback trends and identify areas for improvement.
- Create mechanisms for users to easily report bugs, request features, and offer suggestions for the platform.
- Use the collected data to prioritize development efforts and continuously update the platform based on user needs and feedback.

SLICES/pos: Reproducible Experiment Workflows

Sat, 06 Jan 2024 00:00:00 +0000

SLICES-RI is a european research initiative aiming to create a digital research infrastructure providing an experimental platform for the upcoming decades. One of the main goals of this initiative is the creation of fully reproducible experiments. The SLICES research infrastructure will consist of different experiment sites focusing on different research domains such as AI experiments, Cloud and HPC-driven experiments, or investigations on wireless networks.

To achieve reproducibility, the research group on network architectures and services of the Technical University of Munich develops the SLICES plain orchestrating service (SLICES/pos). This framework supports a fully automated structured experiment workflow. The structure of this workflow acts as a template for the design of experiments. Users that adhere to this template will create inherently reproducible experiments, a feature we call reproducible-by-design.

The SLICES/pos framework currently exists in two versions: (1) A fully-managed pos deployment, that uses the SLICES/pos framework to manage the entire testbed and (2) a hosted SLICES/pos deployment. The hosted SLICES/pos deployment is a temporary deployment that runs inside existing testbeds such as Chameleon or CloudLab.

Additional Information:

plain orchestrating service

Support Additional Programming Languages

Topics: reproducibility, statistics
Skills: Python
Difficulty: Medium
Size: Large (350 hours)
Mentor: Sebastian Gallenmüller, Georg Carle, and Kate Keahey

Design a set of basic examples that demonstrate the usage of pos that can be executed on the SLICES/pos testbed in Munich and the Chameleon testbed. This set of basic examples acts as a demonstration of pos’ capabilities and as a tutorial for new users. Based on these introductory examples, a more complex experiment shall be designed and executed, demonstrating the portability of the experiments between testbeds. This experiment involves the entire experiment workflow consisting of the setup and configuration of the testbed infrastructure, the collection of measurement results, and finally, their evaluation and publication. Multiple results of this experiment shall be created on different testbeds and hardware configurations. The results of the experiments will differ depending on the different hardware platforms on which the experiment was executed. These results shall be evaluated and analyzed to find a common connection between the different result sets of the experiments.

Create introductory examples demonstrating the usage of pos
Design and create a portable complex network experiment based on SLICES/pos
Execute the experiment on different testbeds (Chameleon, SLICES/pos testbed)
Analysis of reproduced experiment
Automated analysis of experimental results
Deduction of a model describing the fundamental connections between different experiment executions