genai | UCSC OSPO

Final Update: Building Intelligent Observability for NRP

Thu, 25 Sep 2025 00:00:00 +0000

I’m excited to share the completion of my OSRE 2025 project, “Intelligent Observability for NRP: A GenAI Approach” and the significant learning journey it has been. We’ve successfully developed a novel InfoAgent architecture that delivers on our core goal: building an ML-powered service for NRP that analyzes monitoring data, detects anomalies, and provides trustworthy GenAI explanations.

How Our Novel InfoAgent Architecture Advances the Observability Mission

Through extensive development and testing, I’ve learned tremendously about building production-ready AI systems and have implemented a novel InfoAgent architecture that orchestrates our specialized agents:

1. Prometheus Metrics Analysis Agent

Function: Continuously ingests and processes NRP’s Prometheus metrics
Progress: Fully implemented data pipelines handling multiple metric types with optimized latency
Purpose: Provides the foundation for anomaly detection by establishing normal behavior baselines

Function: Clarifies ambiguous metrics or patterns before generating explanations
Progress: Completed implementation of Conformal Revision of Questions for disambiguation
Purpose: Ensures explanations address the right system behaviors (e.g., distinguishing CPU saturation from memory pressure)
Deliverable Impact: Successfully improved accuracy of GenAI explanations by eliminating misinterpretations

3. Explanation Generation Agent (AIS)

Function: Creates human-readable explanations and root-cause analysis
Progress: Finalized the Automated Information Seeker with a complete Plan→Validate→Execute→Assess→Revise cycle
Purpose: Transforms technical anomalies into actionable insights for operators
Deliverable Impact: Delivers GenAI explanations with uncertainty quantification

Completed Integration: The Novel InfoAgent Pipeline

We’ve successfully integrated all agents into a unified observability pipeline that represents our novel contribution:

Data Collection: Prometheus metrics → Analysis Agent (comprehensive metrics support)
Anomaly Detection: With statistical confidence bounds using conformal prediction
Query Refinement: Resolving ambiguities before explanation
Explanation Generation: Human-readable analysis with uncertainty awareness
Feedback Loop: System learning from operator interactions (implemented and tested)

Hardware Testing Results

This project taught me valuable lessons about optimizing AI workloads on specialized hardware. We successfully tested our observability framework on Qualcomm Cloud AI 100 Ultra hardware:

Achieved significant performance improvements over baseline CPU implementation
Successfully ported and optimized GLM-4.5 for observability-specific tasks
Validated that specialized AI hardware significantly enhances real-time anomaly detection

Learning Journey and Novel Contributions

Throughout OSRE 2025, I’ve learned extensively about:

Building hierarchical agent coordination systems for complex reasoning
Implementing conformal prediction for trustworthy AI outputs
Creating self-correcting explanation pipelines
Developing adaptive learning systems from operator feedback

The novel InfoAgent architecture demonstrates promising results in our testing environment, with evaluation metrics and benchmarks still being refined as work in progress.

Ongoing Work: Continuing Beyond OSRE

While OSRE 2025 is concluding, I’m actively continuing to contribute to this project:

Preparing the InfoAgent framework for open-source release with comprehensive documentation
Running extended evaluation tests on the Nautilus platform (work in progress)
Writing a research paper detailing our novel architecture
Creating tutorials to help others implement intelligent observability

Project Updates and Code: You can follow my ongoing contributions and access the latest code at https://mreddy10.pages.nrp-nautilus.io/gsocnrp/

Acknowledgments

I’m deeply grateful to my lead mentor Mohammad Firas Sada for his exceptional guidance throughout this transformative learning experience. His insights have been invaluable in helping me develop the novel InfoAgent architecture and navigate the complexities of building production-ready AI systems.

The OSRE 2025 program has been an incredible journey of growth and discovery. I’ve learned not just how to build AI systems, but how to make them trustworthy, explainable, and genuinely useful for real-world operations. The novel InfoAgent architecture we’ve developed serves the original mission: creating an intelligent observability tool that helps NRP operators solve problems faster and keep complex research systems running smoothly.

I’m excited to continue contributing to this project and look forward to seeing how the community adopts and extends these ideas. Check out my contributions and ongoing updates at https://mreddy10.pages.nrp-nautilus.io/gsocnrp/!

Midterm Update: Building Intelligent Observability for NRP

Fri, 01 Aug 2025 00:00:00 +0000

I’m pleased to share the progress we’ve made on my OSRE 2025 project, “Intelligent Observability for Seam: A GenAI Approach” since my initial announcement. We’re working toward our core goal: building an ML-powered service for NRP that analyzes monitoring data, detects anomalies, and provides trustworthy GenAI explanations.

How Our Agents Support the Observability Mission

We’ve been developing specialized agents and tools that work together to support our original project vision:

1. Prometheus Metrics Analysis Agent

Function: Continuously ingests and processes NRP’s Prometheus metrics
Progress: We’ve implemented initial data pipelines for key system metrics
Purpose: Provides the foundation for anomaly detection by establishing normal behavior baselines

Function: Clarifies ambiguous metrics or patterns before generating explanations
Progress: We’ve implemented a basic version of Conformal Revision of Questions to resolve metric ambiguities
Purpose: Aims to ensure explanations address the right system behaviors (e.g., distinguishing CPU saturation from memory pressure)
Deliverable Impact: We hope this will improve accuracy of GenAI explanations by eliminating misinterpretations

3. Explanation Generation Agent (AIS)

Function: Creates human-readable explanations and root-cause analysis
Progress: We’ve built a prototype of the Automated Information Seeker with a Plan→Validate→Execute→Assess→Revise cycle
Purpose: Transforms technical anomalies into actionable insights for operators
Deliverable Impact: Intended to directly deliver on the GenAI explanation component of our tool

Integration Progress

We’re working to connect our agents into a unified observability pipeline:

Data Collection: Prometheus metrics → Analysis Agent
Anomaly Detection: With statistical confidence bounds (in development)
Query Refinement: Resolving ambiguities before explanation
Explanation Generation: Human-readable analysis with uncertainty awareness
Feedback Loop: System learning from operator interactions (planned)

Hardware Testing Opportunity

This project has given us a valuable opportunity to test our observability framework on Qualcomm Cloud AI 100 Ultra hardware. We’re beginning to port different LLM architectures specifically for:

Exploring anomaly detection performance on specialized AI hardware
Testing explanation generation quality across different model architectures
Comparing GLM-4.5 against other models for observability-specific tasks

Next Phase: Completing the Observability Tool

For the remainder of OSRE 2025, we’re focused on:

Finalizing integration of all agents into a cohesive anomaly detection tool with matrix
Validating that our GenAI explanations help operators resolve issues faster for users, which we plan to test on the nautilus matrix platform
Optimizing performance on specialized hardware for NRP’s scale
Preparing the open-source release of our intelligent observability tool

Acknowledgments

I’m deeply grateful to my lead mentor Mohammad Firas Sada for his guidance in keeping our work focused on NRP’s observability needs. His insights have been invaluable in navigating the challenges of this project.

While we’ve developed several agents and frameworks, everything we’re building serves the original mission: creating an intelligent observability tool that helps NRP operators solve problems faster and keep complex research systems running smoothly.

I look forward to sharing more progress on our observability tool with GenAI explanations in the coming weeks!

Vector Embeddings Dataset

Tue, 11 Feb 2025 13:00:00 -0800

Vector Embeddings Dataset

Topics: Vector Embeddings LLMs Transformers
Skills: software development, apis, scripting, python
Difficulty: Moderate
Size: Medium or Large (175 or 350 hours)
Mentors: Jayjeet Chakraborty

To benchmark vector search algorithms (aka ANN algorithms), there are several datasets available but none of them represent actual real world workloads. This is because they usually have small vectors of only a few hundred dimensions. For vector search experiments to represent real world workloads, we want to have datasets with several thousand dimensions like what is generated by OpenAIs text-embedding models. This project aims to create a dataset with 1B embeddings from a wikipedia dataset using open source models. Ideally, we will have 3 versions of this dataset, with 1024, 4096, and 8192 sized embeddings to start with.

genai | UCSC OSPO

Final Update: Building Intelligent Observability for NRP

How Our Novel InfoAgent Architecture Advances the Observability Mission

1. Prometheus Metrics Analysis Agent

2. Query Refinement Agent (CROQ)

3. Explanation Generation Agent (AIS)

Completed Integration: The Novel InfoAgent Pipeline

Hardware Testing Results

Learning Journey and Novel Contributions

Ongoing Work: Continuing Beyond OSRE

Acknowledgments

Midterm Update: Building Intelligent Observability for NRP

How Our Agents Support the Observability Mission

1. Prometheus Metrics Analysis Agent

2. Query Refinement Agent (CROQ)

3. Explanation Generation Agent (AIS)

Integration Progress

Hardware Testing Opportunity

Next Phase: Completing the Observability Tool

Acknowledgments

Vector Embeddings Dataset

Vector Embeddings Dataset