Mark Powers | UCSC OSPO

Assessing and Enhancing CC-Snapshot for Reproducible Experiment Environments

Tue, 18 Feb 2025 00:00:00 +0000

Overview

A critical challenge in computer systems research reproducibility is establishing and sharing experimental environments. While open testbeds like Chameleon provide access to hardware resources, researchers still face significant barriers when attempting to recreate the precise software configurations, dependencies, and system states needed for reproducible experiments. Environment snapshotting tools offer a solution, but face technical challenges in consistently capturing running systems without introducing distortions or requiring disruptive system modifications. This project addresses these fundamental reproducibility barriers by enhancing CC-Snapshot, an tool capturing the experimental environment configured by the user on bare metal images, to create more reliable and consistent system captures that can be shared and redeployed without loss of fidelity.

CC-Snapshot is a tool on the Chameleon testbed that enables users to package their customized environments as complex images or appliances. By allowing researchers to share these environments easily, CC-Snapshot offers a powerful mechanism for reproducibility, ensuring that experiments can be replicated and extended by others.

In this project, you will review existing CC-Snapshot workflows, research the latest snapshotting technologies, and develop enhancements that improve the tool’s usability and reliability. This includes ensuring snapshots are created consistently (even when the OS is actively running), preserving the integrity of user systems, and exploring advanced features such as out-of-band snapshotting and API-based triggers.

Key Outcomes

Improved Snapshot Consistency: New methods to capture the full state of a disk without risking corruption or data inconsistency.
Enhanced Reproducibility: A refined workflow that allows researchers to reliably share custom environments, facilitating collaborative and repeatable experiments.
User-Friendly Tooling: Streamlined processes that reduce disruption to running systems—so installing dependencies or rebooting into special environments is less burdensome.
Exploratory Features (Stretch Goals): Advanced mechanisms to stream disk data in real time during snapshotting and to initiate snapshots via an API call (for parity with VM snapshots).

Topics: Cloud Computing, Systems & Infrastructure, Reproducibility, Operating System Internals

Skills: Linux / OS Concepts, Cloud Tools, Systems Programming / Scripting, DevOps / CI

Difficulty: Moderate

Size: Medium

Mentors: Michael Sherman, Mark Powers

Tasks:

Ensure Snapshot Consistency
- Reboot into a ramdisk and copy the offline disk.
- Use kexec to switch to/from a ramdisk environment without a full reboot.
- Change images to use a snapshot-capable filesystem (e.g., LVM) for safer live snapshots.
- Investigate additional methods (e.g., blog.benjojo.co.uk) for safely imaging live disks.
Prevent System Modifications During Snapshot
- Currently, CC-Snapshot installs dependencies (e.g., qemu-img) on the running system, affecting its state.
- In-Band Fix: Download and run tools in a temp directory with static linking, avoiding system-level changes.
- Out-of-Band Approach: Snapshots done via ramdisk or kexec do not require altering the running system.
API-Triggered Snapshots
- Extend or integrate with the Nova “snapshot instance” API to support the same workflow for bare metal.
- Leverage Ironic’s new “service steps” feature for an automated snapshot pipeline.
(Stretch Goal) Streaming Snapshots
- Modify the workflow to stream data directly to storage, rather than making a full local copy first.
- Explore incremental or differential snapshot techniques to reduce bandwidth usage and storage overhead.

Chameleon Trovi Support for Complex Experiment Appliances

Tue, 18 Feb 2025 00:00:00 +0000

Overview

The discoverability and accessibility of research artifacts remains a significant barrier to reproducibility in computer science research. While digital libraries index research papers, they rarely provide direct access to the artifacts needed to reproduce experiments, especially complex multi-node systems. Additionally, when artifacts are available, they often lack standardized metadata, versioning, and deployment mechanisms that would enable researchers to easily find and reuse them. This project addresses these challenges by extending Trovi, a repository of experimental artifacts executable on open platforms, to support complex, multi-node appliances, making sophisticated experimental environments discoverable, shareable, and deployable through a standardized interface - ultimately lowering the barriers to reproducing complex systems experiments.

Chameleon has historically enabled researchers to orchestrate complex appliances—large, multi-node clusters configured via OpenStack Heat—to conduct advanced experiments. Meanwhile, Chameleon team introduced Trovi as repository for open platforms (beyond Chameleon) that pioneers mechanisms for artifact and platform integration leading to immediate execution for pratical reproducibility. This project aims to bridge the two by adding support in Trovi for importing, discovering, and launching complex appliances. By integrating these capabilities, researchers will be able to one-click deploy complex appliances directly from the Trovi dashboard, archive them for future reference, and reproduce experiments on demand.

Key Outcomes

Extended Trovi API: Enable the import and management of complex appliances as artifacts.
Streamlined One-Click Launch: Integrate with Chameleon’s existing provisioning workflows so users can launch multi-node clusters directly from Trovi.
Enhanced Dashboard Experience: Provide UI assistance for discovering, reviewing, and customizing complex appliance artifacts.
Improved Artifact Reproducibility: Automate the process of exporting CC-snapshot images and other resources to ensure everything is preserved across sites (UC, TACC), highlighting any parameters that need user attention for cross-site portability.

Topics: Reproducible Research, Cloud Computing & Orchestration, OpenStack Heat, UI/UX & Web Development

Skills: Python, APIs, Cloud (OpenStack), DevOps & Automation, Frontend

Difficulty: Hard

Size: Large

Mentors: Mark Powers

Tasks:

Extensions to the Trovi API
- Add support for importing complex appliances as artifacts (including Heat templates, metadata, and associated disk images).
- Develop methods for tagging, versioning, and categorizing these appliances, making them easier to discover.
One-Click Launch of Complex Appliances
- Integrate with Chameleon’s orchestration engine, enabling single-click cluster deployments from the Trovi UI.
- Validate correct configuration and resource availability through automated checks.
Trovi Dashboard Enhancements
- Update the front-end to provide intuitive controls for customizing or parameterizing complex appliances before launching.
- Offer a clear workflow for reviewing dependencies, resource requirements, and usage instructions.
Automated Export & Multi-Site Testing
- Streamline the export of snapshots or images into Trovi as part of the appliance import process.
- Optionally re-run the imported appliances at multiple sites (UC, TACC), detecting any unparameterized settings or missing dependencies.

Widgets for Python-chi in Jupyter

Tue, 18 Feb 2025 00:00:00 +0000

Overview

Reproducibility challenges in research extend beyond code and environments to the experimental workflow itself. When experiments involve dynamic resource allocation, monitoring, and reconfiguration, researchers often struggle to document these interactive steps in a way that others can precisely follow. The lack of structured workflow documentation and real-time feedback creates barriers for reviewers attempting to reproduce experiments, as they cannot easily verify whether their resource configurations match the original experiment’s state. This project addresses these challenges by developing interactive Jupyter widgets that make experiment resource management more visual, intuitive, and self-documenting—transforming ad-hoc command sequences into reproducible workflows that automatically log interactions and configuration changes while providing immediate visual feedback on experiment topology and resource states.

As cloud researchers often work with Jupyter Notebooks for interactive data analysis and experimentation, the python-chi library offers a powerful way to automate and control resources on Chameleon Cloud. This project will extend python-chi by adding interactive widgets specifically designed for use in Jupyter, empowering users to launch, monitor, and manage their experiments without leaving the notebook environment. By bringing visual and intuitive controls directly into the user’s workflow, we aim to improve both reproducibility and usability for complex resource management tasks.

Key Outcomes

User-Friendly Jupyter Widgets: Develop a suite of widgets to visualize reserved resources, hardware availability, and experiment topologies in real time.
Integrated Experiment Management: Enable researchers to orchestrate experiments (launch, configure, monitor) within a single, notebook-centric workflow.
Enhanced Feedback & Usability: Provide clear, asynchronous status updates and resource reconfiguration progress, reducing confusion and user error.
Improved Reproducibility: By automating and logging widget interactions, experiments become more traceable and easier to replicate.

Topics: Interactive Data Tools, Cloud Resource Management, DevOps & Automation, User Experience (UX)

Skills:

Python & Jupyter: Experience creating custom Jupyter widgets, using ipywidgets or similar frameworks.
Cloud Automation: Familiarity with how resources are provisioned, monitored, and deprovisioned on Chameleon.
Frontend / GUI Development: Basic understanding of web technologies (HTML/CSS/JavaScript) can be helpful for widget design.
Software Engineering & CI: Ability to version-control, test, and deploy Python packages.

Difficulty: Moderate

Size: Medium

Mentor: Michael Sherman, Mark Powers

Tasks:

Resource Visualization Widgets
- Build custom widgets that show reserved resources (nodes, networks, storage) in Jupyter.
- Provide an interactive topology view for experiments, indicating node statuses and connections.
Experiment Setup & Execution
- Add controls for launching and managing experiments directly from notebooks.
- Show feedback (e.g., progress bars, status messages) as resources are being allocated or reconfigured.
Hardware Availability & Status Tracking
- Implement a widget that provides real-time data on Chameleon’s hardware availability (bare metal, VMs, GPU nodes, etc.).
- Allow users to filter or select specific resources based on current hardware states.
Usability & Feedback Loop
- Gather user feedback on the widget designs and workflows.
- Refine the interface to minimize clicks, improve clarity, and reduce friction for common tasks.

Chameleon Trovi Redesign

Wed, 21 Feb 2024 13:43:55 -0600

Trovi on Chameleon is an open-source service designed to significantly enhance the practical reproducibility of computer science research. By allowing Chameleon users to upload, share, and access packaged experiments and other research artifacts, Trovi aims to streamline the process of replicating and building upon existing studies. This capability is crucial in the scientific community, where the ability to accurately reproduce research results is as fundamental to validating, critiquing, and extending scientific findings as reading papers. The importance of Trovi lies in its potential to serve as a centralized hub that facilitates the exchange of valuable research outputs, promotes transparency, and fosters collaboration among researchers. By improving the ease with which experiments can be replicated and data can be shared, Trovi supports the advancement of knowledge and innovation in the field of computer science, making it an essential tool for researchers seeking to contribute to the development of reproducible and robust scientific research.

This project will focus on the evolution of Trovi. It will aim to enhance Trovi as a tool to advance practical reproducibility in CS research. Students will evaluate the most important use cases and enabling features necessary to enhance Trovi’s functionality and user experience. With these design insights, students will then create a robust interface that allows researchers to integrate experiment code and data easily as packaged artifacts, similar to the user-friendly design of Google Colab, and build off other users’ artifacts to create novel experiments, similar to the design of GitHub. Furthermore, students will create comprehensive documentation with valuable insights into what works well and what requires improvement, creating a dynamic feedback loop to guide the ongoing redesign process. Lastly, students will actively participate in designing webinars, creating and posting video tutorials, and organizing academic events at the University of Chicago to showcase the work on Trovi. This multifaceted project ensures a well-rounded experience and fosters a collaborative learning environment.

Each of the project ideas below focuses on a different aspect of the overall goal to enhance Trovi as a tool for advancing practical reproducibility in CS research. They are designed to offer a comprehensive approach, from technical development to community engagement, ensuring a well-rounded enhancement of the service.

Topics: User Interface Design User Experience Web Development
Skills: HTML/CSS, JavaScript, UX design principles
Difficulty: Moderate to Hard
Size: Medium to Large
Mentors: Mark Powers
Tasks:
- Conduct user research to understand the needs and pain points of current and potential Trovi users.
- Design wireframes and prototypes that incorporate user feedback and aim to simplify the process of uploading, sharing, and reusing research artifacts.
- Implement the frontend redesign using a modern web framework to ensure responsiveness and ease of use.

Packaged Artifacts Integration System

Topics: Cloud Computing Data Management Web APIs
Skills: Python, RESTful APIs, Docker, Git
Difficulty: Hard
Size: Large
Mentors: Mark Powers
Tasks:
- Develop a system that allows users to easily package and upload their experimental code and data to Trovi.
- Create a standardized format or set of guidelines for packaging experiments to ensure consistency and ease of use.
- Implement API endpoints that enable automated uploads, downloads, and integration with other tools like GitHub or Zenodo.
- Test the system with real-world experiments to ensure reliability and ease of integration.

Community Engagement and Educational Materials

Topics: Educational Technology Community Building Content Creation
Skills: Video Editing, Public Speaking, Event Planning
Difficulty: Moderate
Size: Medium
Mentors: Mark Powers
Tasks:
- Design and organize webinars that introduce Trovi and its new features to the research community.
- Create engaging video tutorials that guide users through the process of using Trovi for their research needs.
- Develop comprehensive documentation that covers both basic and advanced use cases, troubleshooting, and tips for effective collaboration using Trovi.
- Organize academic events, such as workshops or hackathons, that encourage the use of Trovi for collaborative research projects.

Feedback Loop and Continuous Improvement System

Topics: Software Engineering Data Analysis User Feedback
Skills: Python, SQL, Data Visualization, Web Development
Difficulty: Moderate
Size: Medium
Mentors: Mark Powers
Tasks:
- Implement a system within Trovi for collecting, storing, and analyzing user feedback and usage data.
- Develop dashboards that visualize feedback trends and identify areas for improvement.
- Create mechanisms for users to easily report bugs, request features, and offer suggestions for the platform.
- Use the collected data to prioritize development efforts and continuously update the platform based on user needs and feedback.

Mark Powers | UCSC OSPO

Assessing and Enhancing CC-Snapshot for Reproducible Experiment Environments

Overview

Key Outcomes

Chameleon Trovi Support for Complex Experiment Appliances

Overview

Key Outcomes

Widgets for Python-chi in Jupyter

Overview

Key Outcomes

Chameleon Trovi Redesign

User Interface Redesign for Experiment Artifacts Sharing

Packaged Artifacts Integration System

Community Engagement and Educational Materials

Feedback Loop and Continuous Improvement System