Final Update: Building Intelligent Observability for NRP

Thu, 25 Sep 2025 00:00:00 +0000

I’m excited to share the completion of my OSRE 2025 project, “Intelligent Observability for NRP: A GenAI Approach” and the significant learning journey it has been. We’ve successfully developed a novel InfoAgent architecture that delivers on our core goal: building an ML-powered service for NRP that analyzes monitoring data, detects anomalies, and provides trustworthy GenAI explanations.

How Our Novel InfoAgent Architecture Advances the Observability Mission

Through extensive development and testing, I’ve learned tremendously about building production-ready AI systems and have implemented a novel InfoAgent architecture that orchestrates our specialized agents:

1. Prometheus Metrics Analysis Agent

Function: Continuously ingests and processes NRP’s Prometheus metrics
Progress: Fully implemented data pipelines handling multiple metric types with optimized latency
Purpose: Provides the foundation for anomaly detection by establishing normal behavior baselines

Function: Clarifies ambiguous metrics or patterns before generating explanations
Progress: Completed implementation of Conformal Revision of Questions for disambiguation
Purpose: Ensures explanations address the right system behaviors (e.g., distinguishing CPU saturation from memory pressure)
Deliverable Impact: Successfully improved accuracy of GenAI explanations by eliminating misinterpretations

3. Explanation Generation Agent (AIS)

Function: Creates human-readable explanations and root-cause analysis
Progress: Finalized the Automated Information Seeker with a complete Plan→Validate→Execute→Assess→Revise cycle
Purpose: Transforms technical anomalies into actionable insights for operators
Deliverable Impact: Delivers GenAI explanations with uncertainty quantification

Completed Integration: The Novel InfoAgent Pipeline

We’ve successfully integrated all agents into a unified observability pipeline that represents our novel contribution:

Data Collection: Prometheus metrics → Analysis Agent (comprehensive metrics support)
Anomaly Detection: With statistical confidence bounds using conformal prediction
Query Refinement: Resolving ambiguities before explanation
Explanation Generation: Human-readable analysis with uncertainty awareness
Feedback Loop: System learning from operator interactions (implemented and tested)

Hardware Testing Results

This project taught me valuable lessons about optimizing AI workloads on specialized hardware. We successfully tested our observability framework on Qualcomm Cloud AI 100 Ultra hardware:

Achieved significant performance improvements over baseline CPU implementation
Successfully ported and optimized GLM-4.5 for observability-specific tasks
Validated that specialized AI hardware significantly enhances real-time anomaly detection

Learning Journey and Novel Contributions

Throughout OSRE 2025, I’ve learned extensively about:

Building hierarchical agent coordination systems for complex reasoning
Implementing conformal prediction for trustworthy AI outputs
Creating self-correcting explanation pipelines
Developing adaptive learning systems from operator feedback

The novel InfoAgent architecture demonstrates promising results in our testing environment, with evaluation metrics and benchmarks still being refined as work in progress.

Ongoing Work: Continuing Beyond OSRE

While OSRE 2025 is concluding, I’m actively continuing to contribute to this project:

Preparing the InfoAgent framework for open-source release with comprehensive documentation
Running extended evaluation tests on the Nautilus platform (work in progress)
Writing a research paper detailing our novel architecture
Creating tutorials to help others implement intelligent observability

Project Updates and Code: You can follow my ongoing contributions and access the latest code at https://mreddy10.pages.nrp-nautilus.io/gsocnrp/

Acknowledgments

I’m deeply grateful to my lead mentor Mohammad Firas Sada for his exceptional guidance throughout this transformative learning experience. His insights have been invaluable in helping me develop the novel InfoAgent architecture and navigate the complexities of building production-ready AI systems.

The OSRE 2025 program has been an incredible journey of growth and discovery. I’ve learned not just how to build AI systems, but how to make them trustworthy, explainable, and genuinely useful for real-world operations. The novel InfoAgent architecture we’ve developed serves the original mission: creating an intelligent observability tool that helps NRP operators solve problems faster and keep complex research systems running smoothly.

I’m excited to continue contributing to this project and look forward to seeing how the community adopts and extends these ideas. Check out my contributions and ongoing updates at https://mreddy10.pages.nrp-nautilus.io/gsocnrp/!

final | UCSC OSPO