computational pathology | UCSC OSPO

Omni-ST: Instruction-Driven Any-to-Any Multimodal Modeling for Spatial Transcriptomics

Thu, 29 Jan 2026 00:00:00 +0000

Project description

Spatial transcriptomics (ST) integrates spatially resolved gene expression with tissue morphology, enabling the study of cellular organization, tissue architecture, and disease microenvironments. Modern ST datasets are inherently multimodal, combining histology images (H&E / IF), gene expression vectors, spatial graphs, cell annotations, and free-text pathology descriptions.

However, most existing ST methods are task-specific and modality-siloed: separate models are trained for image-to-gene prediction, spatial domain identification, cell type classification, or text-based interpretation. This fragmentation limits cross-task generalization and scalability.

Omni-ST proposes a single instruction-driven any-to-any multimodal backbone that treats each spatial transcriptomics modality as a “language” and formulates all tasks as:

Instruction + Input Modality → Output Modality

Natural language is elevated from auxiliary metadata to a unifying interface that specifies task intent, target modality, and biological context. This paradigm enables flexible, interpretable, and extensible spatial reasoning within a single model.

Project Idea: Instruction-Driven Any-to-Any Modeling for Spatial Transcriptomics

Topics: spatial transcriptomics, multimodal learning, instruction tuning, computational pathology
Skills: PyTorch, deep learning, Transformers, multimodal representation learning
Difficulty: Hard
Size: 350 hours

Mentor:

Xi Li — mailto:xil43@uci.edu

Essential information:

Design a unified multimodal backbone with lightweight modality adapters for histology images, gene expression vectors, spatial graphs, and text.
Use natural language instructions to condition model behavior, enabling any-to-any translation without task-specific heads.
Support core tasks including image → gene expression prediction, gene expression → cell type / spatial domain identification, region → text-based biological explanation, and text-based spatial retrieval.
Evaluate the model across multiple spatial transcriptomics tasks within a single framework, emphasizing generalization and interpretability.
Develop visualization and interpretation tools such as spatial maps and language-grounded explanations.

Expected deliverables:

An open-source PyTorch implementation of the Omni-ST framework.
Unified multitask benchmarks for spatial transcriptomics.
Visualization and interpretation tools for spatial predictions.
Documentation and tutorials demonstrating how to add new tasks via instructions.

HistoMoE: A Histology-Guided Mixture-of-Experts Framework for Gene Expression Prediction

Tue, 20 Jan 2026 00:00:00 +0000

Topics: computational pathology, spatial transcriptomics, gene expression prediction, mixture-of-experts, multimodal learning
Skills:
- Programming Languages: Python; experience with PyTorch preferred
- Machine Learning: CNNs / vision encoders, mixture-of-experts, multimodal representation learning
- Data Analysis: handling large-scale histology image patches and gene expression matrices
- Bioinformatics Knowledge (preferred): familiarity with spatial transcriptomics or scRNA-seq data
Difficulty: Advanced
Size: Large (350 hours)
Mentors: Ziheng Duan (contact person)

Project Idea Description

Histology imaging is one of the most widely available data modalities in biomedical research and clinical practice, capturing rich morphological information about tissues and disease states. In parallel, spatial transcriptomics (ST) technologies provide spatially resolved gene expression measurements, enabling unprecedented insights into tissue organization and cellular heterogeneity. However, the high cost and limited accessibility of ST experiments remain a major barrier to their widespread adoption.

Predicting gene expression directly from histology images offers a promising alternative, enabling molecular-level inference from routinely collected pathology data. Existing approaches typically rely on a single global model that maps image embeddings to gene expression profiles. While effective to some extent, these models struggle to capture the strong organ-, tissue-, and cancer-specific heterogeneity that underlies gene expression patterns.

This project proposes HistoMoE, a histology-guided mixture-of-experts (MoE) framework that explicitly models biological heterogeneity by learning specialized expert models for different cancer types or organs, and dynamically routing histology image patches to the most relevant experts.

Key Idea and Technical Approach

As illustrated in the figure above, HistoMoE integrates multiple data modalities and learning components:

Vision Encoder
Histology image patches are encoded into high-dimensional visual representations using a convolutional or transformer-based vision backbone.
Text / Metadata Encoder
Sample-level metadata (e.g., tissue type, organ, disease context) is encoded using a lightweight text or embedding model.
Gating Network
A gating network jointly considers image and metadata embeddings to infer routing weights over multiple cancer- or organ-specific expert models.
Expert Models
Each expert specializes in modeling gene expression patterns for a specific biological context (e.g., CCRCC, COAD, LUAD), producing patch-level gene expression predictions.

By explicitly modeling biological structure through expert specialization, HistoMoE aims to improve both prediction accuracy and interpretability, allowing researchers to understand which biological experts drive each prediction.

Project Objectives

Design and Implement the HistoMoE Framework
- Build a modular MoE architecture with pluggable vision encoders, gating networks, and expert models.
Multimodal Routing and Expert Specialization
- Explore how image features and metadata jointly inform expert selection.
Benchmarking and Evaluation
- Compare HistoMoE against single-model baselines on multiple cancer and organ-specific spatial transcriptomics datasets.
Interpretability Analysis
- Analyze expert routing behavior to reveal biologically meaningful patterns.

Project Deliverables

Open-Source HistoMoE Codebase
- Well-documented Python implementation with training, evaluation, and visualization tools.
Benchmark Results
- Quantitative comparisons demonstrating improvements over non-expert baselines.
Visualization and Analysis Tools
- Tools for inspecting expert usage, routing weights, and gene-level predictions.
Documentation and Tutorials
- Clear instructions and examples to enable adoption by the research community.

Impact

HistoMoE introduces an expert-system perspective to histology-based gene expression prediction, bridging morphological and molecular representations through biologically informed specialization. By combining multimodal learning with mixture-of-experts modeling, this project advances the interpretability and accuracy of computational pathology methods and contributes toward scalable, cost-effective alternatives to spatial transcriptomics experiments.

Disentangled Generation and Editing of Pathology Images

Fri, 07 Feb 2025 00:00:00 +0000

Topics: computational pathology, image generation, disentangled representations, latent space manipulation, deep learning
Skills:
- Programming Languages:
  - Proficient in Python, with experience in machine learning libraries such as PyTorch or TensorFlow.
- Generative Models:
  - Familiarity with Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), and contrastive learning methods.
- Data Analysis:
  - Image processing techniques, statistical analysis, and working with histopathology datasets.
- Biomedical Knowledge (preferred):
  - Basic understanding of histology, cancer pathology, and biological image annotation.
Difficulty: Advanced
Size: Large (350 hours). The project involves substantial computational work, model development, and evaluation of generated pathology images.
Mentors: Xi Li (contact person), Mentor Name

Project Idea Description

The project aims to advance the generation and disentanglement of pathology images, focusing on precise control over key histological features. By leveraging generative models, we seek to create synthetic histological images where specific pathological characteristics can be independently controlled.

Challenges in Current Approaches

Current methods in histopathology image generation often struggle with:

Feature Entanglement: Difficulty in isolating individual factors such as cancer presence, severity, or staining variations.
Lack of Control: Limited capability to manipulate specific pathological attributes without affecting unrelated features.
Consistency Issues: Generated images often fail to maintain realistic cellular distributions, affecting biological validity.

Project Motivation

This project proposes a disentangled representation framework to address these limitations. By separating key features within the latent space, we aim to:

Control Histological Features: Adjust factors such as cancer presence, tumor grade, number of malignant cells, and staining methods.
Ensure Spatial Consistency: Maintain the natural distribution of cells during image reconstruction and editing.
Enable Latent Space Manipulation: Provide interpretable controls for editing and generating realistic histopathology images.

Project Objectives

Disentangled Representation Learning:
- Develop generative models (e.g., VAEs, GANs) to separate and control histological features.
Latent Space Manipulation:
- Design mechanisms for intuitive editing of pathology images through latent space adjustments.
Spatial Consistency Validation:
- Implement evaluation metrics to ensure that cell distribution remains biologically consistent during image generation.

Project Deliverables

Generative Model Framework:
- An open-source Python implementation for pathology image generation and editing.
Disentangled Latent Space Tools:
- Tools for visualizing and manipulating latent spaces to control specific pathological features.
Evaluation Metrics:
- Comprehensive benchmarks assessing image quality, feature disentanglement, and biological realism.
Documentation and Tutorials:
- Clear guidelines and code examples for the research community to adopt and build upon this work.

Impact

By enabling precise control over generated histology images, this project will contribute to data augmentation, model interpretability, and biological insight in computational pathology. The disentangled approach offers new opportunities for researchers to explore disease mechanisms, develop robust diagnostic models, and improve our understanding of cancer progression and tissue morphology.