Yuyang (Roy) Huang | UCSC OSPO

FEP-Bench: Benchmarking for Enhanced Feature Engineering and Preprocessing in Machine Learning

Mon, 03 Jun 2024 00:00:00 +0000

Project Idea Description

Topics: Storage systems, machine learning
Skills: Python, PyTorch, Bash scripting, Linux, Machine Learning modeling
Difficulty: Hard
Size: Large (350 hours)
Mentors: Yuyang (Roy) Huang (primary contact), Swami Sundararaman
Contributor(s): Lihaowen (Jayce) Zhu

In the realm of machine learning (ML), preprocessing of data is a critical yet often underappreciated phase, consuming approximately 80% of the time in common ML tasks. This extensive time consumption can be attributed to various challenges encountered from both data and computation perspectives.

From the data side, one significant challenge is the slow retrieval of data from data lakes, which are storage repositories that hold a vast amount of raw data in its native format. However, the process of extracting this data can be slow, causing computation cycles to wait for data arrival and leading to delays in the entire preprocessing phase. Furthermore, the size of the data often exceeds the memory capacity of standard computing systems. This is a frequent occurrence in ML, as datasets are typically large and complex. Handling such large datasets requires sophisticated memory management techniques to ensure efficient preprocessing without overwhelming the system’s memory.

On the computation side, a naive solution to data operations, especially aggregation, often leads to inefficiencies. These operations may require grouping a large chunk of data as a prerequisite before performing any actual computation. This grouping, without careful configuration and management, can trigger serious data shuffling, leading to extensive remote data movement when the data is distributed across various storage systems. Such data movement is not only time-consuming but also resource-intensive.

To mitigate these challenges, there is a pressing need to design better caching, prefetching, and heuristic strategies for data preprocessing. The team aims to significantly reduce the time and resources required for preprocessing by optimizing data retrieval and computational processes.

However, prior to the design and implementation of such a system, a systematic understanding of the preprocessing workflow is essential. Hence, throughout the program, the students will need to:

Understand the current system used to preprocess data for ML training, for example, Hadoop or Spark.
Collect the common datasets used for different types of ML models.
Collect the typical operations used for preprocessing these datasets.
Benchmark the performance in these operations under the existing frameworks under various experimental settings.
Package the benchmark such that the team can later use it for reproduction or evaluation.

Project Deliverable

A rolodex for the commonly used dataset and corresponding preprocess operations and expected output formats/types
A Chameleon Trovi package that preprocess the dataset with single-machine preprocessing framework like pandas
A Chameleon Trovi package that preprocess the dataset in an existing distributed computation framework like Hadoop or Spark

EdgeRep: Reproducing and benchmarking edge analytic systems

Fri, 02 Feb 2024 00:00:00 +0000

Topics: video analytics, machine learning
Skills: Python, PyTorch, Bash scripting, Linux, Machine Learning modeling
Difficulty: Medium
Size: Large (350 hours)
Mentors: Yuyang (Roy) Huang (contact person), Junchen Jiang

Project Idea Description

With the flourishing of ideas like smart cities and smart manufacturing, a massive number of edge devices (e.g., traffic or security cameras, thermometers, flood sensors, etc.) are deployed and connected to the network. These devices collect and analyze data across space and time, aiding stakeholders like city governments and manufacturers in optimizing their plans and operations. However, the sheer number of edge devices and the large amount of communication among the devices and central servers raises significant challenges in how to manage and schedule resources. This includes network bandwidth between the devices and computing power on both edge devices and bare metal servers, all to maintain the reliable service capability of running applications.

Moreover, given the limited resources available to edge devices, there’s an emerging trend to reduce average compute and/or bandwidth usage. This is achieved by leveraging the uneven distribution of interesting events with respect to both time and space in the input data. This, in turn, introduces further challenges in provisioning and managing the amount of resources available to edge devices. The resource demands of running applications can greatly depend on the input data, which is both dynamic and unpredictable.

Keeping these challenges in mind, the team previously designed and implemented a dynamic resource manager capable of understanding the applications and making decisions based on this understanding at runtime. However, such a resource manager has only been tested with a limited number and types of video analytic applications. Thus, through the OSRE24 project, we aim to:

Collect a wide range of videos to form a comprehensive video dataset
Reproduce other state-of-art self-adaptive video analytic applications
Package the dataset as well as the application to publish them on Chameleon Trovi site

Project Deliverable

Collect a wide range of videos to form a comprehensive video dataset
Reproduce other state-of-art self-adaptive video analytic applications
Package the dataset as well as the application to publish them on Chameleon Trovi site

FEP-Bench: Benchmarks for understanding featuring engineering and preprocessing bottlenecks

Fri, 02 Feb 2024 00:00:00 +0000

Topics: storage system, scheduling, distributed system, machine learning
Skills: Python, PyTorch, Bash scripting, Linux, Machine Learning modeling
Difficulty: Hard
Size: Large (350 hours)
Mentors: Yuyang (Roy) Huang (contact person), Swami Sundararaman

Project Idea Description

However, prior to the design and implementation of such a system, a systematic understanding of the preprocessing workflow is essential. Hence, throughout the program, the students will need to:

Understand the current system used to preprocess data for ML training, for example, Hadoop or Spark.
Collect the common datasets used for different types of ML models.
Collect the typical operations used for preprocessing these datasets.
Benchmark the performance in these operations under the existing frameworks under various experimental settings.
Package the benchmark such that the team can later use it for reproduction or evaluation.

Project Deliverable

Understand the current system used to preprocess data for ML training, for example, Hadoop or Spark.
Collect the common datasets used for different types of ML models.
Collect the typical operations used for preprocessing these datasets.
Benchmark the performance in these operations under the existing frameworks under various experimental settings.
Package the benchmark such that the team can later use it for reproduction or evaluation.