indexing | UCSC OSPO

End-term Blog: StatWrap: Cross-Project Searching and Classification using Local Indexing

Sat, 23 Aug 2025 00:00:00 +0000

Introduction

Hello everyone!
I am Debangi Ghosh from India, an undergraduate student at the Indian Institute of Technology (IIT) BHU, Varanasi. As part of the StatWrap: Cross-Project Searching and Classification using Local Indexing project, my proposal, under the mentorship of Luke Rasmussen, focuses on developing a full-text search service within the StatWrap user interface. This involves evaluating different search libraries and implementing a classification system to distinguish between active and past projects.

About the Project

As part of the project, I am working on enhancing the usability of StatWrap by enabling efficient cross-project search capabilities. The goal is to make it easier for researchers to discover relevant projects, notes, and assets across both current and archived work, using information that is either user-entered or passively collected by StatWrap.

Given the sensitivity of the data involved, one of the key requirements is that all indexing and search operations must be performed locally. To address this, my responsibilities include:

Evaluating open-source search libraries suitable for local indexing and retrieval
Building the full-text search functionality directly into the StatWrap UI to allow seamless querying across projects
Ensuring reliability through the development of unit tests and comprehensive system testing
Implementing a classification system to label projects as “Active,” “Pinned,” or “Past” within the user interface

This project offers a great opportunity to work at the intersection of software development, information retrieval, and user-centric design—while contributing to research reproducibility and collaboration within scientific workflows.

Deliverables

The project has reached the end of its scope after 12 weeks of work. Here’s a breakdown:

1. Descriptive Comparison of Open-Source Libraries

Compared various open-source search libraries based on evaluation criteria such as indexing speed, search speed, memory usage, typo tolerance, fuzzy searching, partial matching, full-text queries, contextual search, Boolean support, exact word match, installation ease, maintenance, documentation, and developer experience. Decided upon the weights to assign to each of the features and point out the best library to use. According to our weights assigned,

These results are after tuning the hyperparameters to give the best set of results For huge data, FlexSearch has the least memory usage, followed by MiniSearch. The examples we used were limited, so Minisearch had the better memory usage results. Along with the research and evaluation, I looked upon the Performance Benchmark of Full-Text-Search Libraries (Stress Test), available here

The benchmark was measured in terms per seconds, higher values are better (except the test “Memory”). The memory value refers to the amount of memory which was additionally allocated during search.

FlexSearch performs queries up to 1,000,000 times faster compared to other libraries by also providing powerful search capabilities like multi-field search (document search), phonetic transformations, partial matching, tag-search, result highlighting or suggestions. Bigger workloads are scalable through workers to perform any updates or queries to the index in parallel through dedicated balanced threads.

2. The Search User Interface

3. Complete Search Execution Pipeline

4. FlexSearch Features

1. Persistent Indexing with Automatic Loading

Index persistence: Search index automatically saves to disk and loads on startup
Fast restoration: Rebuilds FlexSearch indices from saved document store without re-scanning files
Incremental updates: Detects project changes and updates only modified content
Background processing: Index updates happen asynchronously without blocking the User Interface.

2. Multi-Document Type Support

Unified search: Single search interface for projects, files, people, notes, and assets
Type-specific indices: Separate FlexSearch indices optimized for each document type
Cross-reference capabilities: Documents can reference and link to each other
Flexible schema: Each document type has tailored fields for optimal search performance

3. Intelligent File Content Indexing

Configurable file size limits: Admin-controlled maximum file size for content indexing
Smart file detection: Automatically identifies text files by extension and filename patterns
Content extraction: Full-text indexing with snippet generation for search results
Performance optimization: Skips binary files and respects size constraints to maintain speed

4. Advanced Query Processing

Multi-strategy search: Combines exact matches, fuzzy search, partial matches, and contextual search
Query preprocessing: Removes stop words and applies linguistic filters
Relevance scoring: Custom scoring algorithm considering multiple factors:
- Exact phrase matches (highest weight)
- Individual word matches
- Term frequency with logarithmic capping
- Position-based scoring (earlier matches rank higher)
- Proximity bonuses for terms appearing near each other
- Completeness penalties for missing query terms

5. Real-Time Search Suggestions

Autocomplete support: Dynamic suggestions based on indexed document titles
Search history: Maintains recent searches for quick re-execution
Debounced input: Prevents excessive API calls during typing
Contextual suggestions: Suggestions adapt based on current filters and context

6. Comprehensive Filtering System

Type filtering: Filter by document type (projects, files, people, etc.)
Project scoping: Limit searches to specific projects
File type filtering: Filter files by extension
Advanced search panel: Collapsible interface for power users
Filter persistence: Maintains filter state across searches

7. Performance Monitoring & Analytics

Real-time metrics: Track search times, cache hit rates, and index statistics
Performance dashboard: Visual indicators for system health
Cache management: LRU cache with configurable size and TTL
Search analytics: Historical data on search patterns and performance

8. Index Management Tools

Export/Import functionality: Backup and restore search indices
Full reindexing: Complete index rebuild with progress tracking
Index deletion: Clean slate functionality for troubleshooting
File size adjustment: Modify indexing constraints and rebuild affected content
Index statistics: Detailed breakdown of indexed content by type and project

9. Robust Error Handling & Resilience

Graceful degradation: System continues operating even with partial index corruption
File system error handling: Handles missing files, permission issues, and path changes
Memory management: Prevents memory leaks during large indexing operations
Recovery mechanisms: Automatic fallback to basic search if advanced features fail

10. User Experience Enhancements

Keyboard shortcuts: Ctrl+K to focus search, Escape to clear
Result highlighting: Visual emphasis on matching terms in results
Expandable results: Drill down into detailed information for each result
Loading states: Clear feedback during indexing and search operations
Responsive tabs: Organized results by type with badge counts

5. Classification of Active and Past Projects

A classification system is added within the User Interface similar to “Add to Favorites” option. A new project added by default moves to “Active” section, unless explicitely marked as “Past”. Similarly, when a project is unpinned from Favorites, it goes to “Active” Section.

Conclusion and future Scope

Building a comprehensive search system requires careful attention to performance, user experience, and maintainability. FlexSearch provided the foundation, but the real value came from thoughtful implementation of persistent indexing, advanced scoring, and robust error handling. The result is a search system that feels instant to users while handling complex queries across diverse document types.

The key to success was treating search not as a single feature, but as a complete subsystem with its own data management, performance monitoring, and user interface considerations. By investing in these supporting systems, the search functionality became a central, reliable part of the application that users can depend on.

The future scope would include:

Using a database (for example, SQLite), instead of JSON, which is better for this use case than JSON due to better and efficient query performance and atomic (CRUD) operations.
Integrating any suggestions from my mentors, as well as improvements we feel are necessary.
Developing unit tests for further functionalities and improvements.

Mid-term Blog: StatWrap: Cross-Project Searching and Classification using Local Indexing

Tue, 15 Jul 2025 00:00:00 +0000

Introduction

About the Project

As part of the project, I am working on enhancing the usability of StatWrap by enabling efficient cross-project search capabilities. The goal is to make it easier for investigators to discover relevant projects, notes, and assets—across both current and archived work—using information that is either user-entered or passively collected by StatWrap.

Given the sensitivity of the data involved, one of the key requirements is that all indexing and search operations must be performed locally. To address this, my responsibilities include:

Evaluating open-source search libraries suitable for local indexing and retrieval
Building the full-text search functionality directly into the StatWrap UI to allow seamless querying across projects
Ensuring reliability through the development of unit tests and comprehensive system testing
Implementing a classification system to label projects as “Active,” “Pinned,” or “Past” within the user interface

Progress

It has been more than six weeks since the project began, and significant progress has been made. Here’s a breakdown:

1. Descriptive Comparison of Open-Source Libraries

2. The Libraries

Lunr.js
A small, client-side full-text search engine that mimics Solr capabilities.
- Field-based search, boosting
- Supports TF-IDF, inverted index
- No built-in fuzzy search (only basic wildcards)
- Can serialize/deserialize index
- Not designed for large datasets
- Moderate memory usage and indexing speed
- Good documentation
- Best for: Static websites or SPAs needing simple in-browser search
ElasticLunr.js
A lightweight, more flexible alternative to Lunr.js.
- Dynamic index (add/remove docs)
- Field-based and weighted search
- No advanced fuzzy matching
- Faster and more customizable than Lunr
- Smaller footprint
- Easy to use and maintain
- Best for: Developers wanting Lunr-like features with simpler customization
Fuse.js
A fuzzy search library ideal for small to medium datasets.
- Fuzzy search with typo tolerance
- Deep key/path searching
- No need to build index
- Highly configurable (threshold, distance, etc.)
- Linear scan = slower on large datasets
- Not full-text search (scoring-based match)
- Extremely easy to set up and use
- Best for: Fuzzy search in small in-memory arrays (e.g., auto-suggest, dropdown filters)
FlexSearch
A blazing-fast, modular search engine with advanced indexing options.
- Extremely fast search and indexing
- Supports phonetic, typo-tolerant, and partial matching
- Asynchronous support
- Multi-language + Unicode-friendly
- Low memory footprint
- Configuration can be complex for beginners
- Best for: High-performance search in large/multilingual datasets
MiniSearch
A small, full-text search engine with balanced performance and simplicity.
- Fast indexing and searching
- Fuzzy search, stemming, stop words
- Field boosting and prefix search
- Compact, can serialize index
- Clean and modern API
- Lightweight and easy to maintain
- Best for: Balanced, in-browser full-text search for moderate datasets
Search-Index
A persistent, full-featured search engine for Node.js and browsers.
- Persistent storage with LevelDB
- Real-time indexing
- Fielded queries, faceting, filtering
- Advanced queries (Boolean, range, etc.)
- Slightly heavier setup
- Good for offline/local-first apps
- Browser usage more complex than others
- Best for: Node.js apps, not directly compatible with the Electron + React environment of StatWrap

3. Developer Experience and Maintenance

We analyzed the download trends of the search libraries using npm trends, and also reviewed their maintenance statistics to assess how frequently they are updated.

4. Comparative Analysis After Testing

Each search library was benchmarked against a predefined set of queries based on the same evaluation criteria.
We are yet to finalize the weights for each criterion, which will be done during the end-term evaluation.

5. The User Interface

The user interface includes options to search using three search modes (Basic, Advanced, Boolean operators) with configurable parameters. Results are sorted based on relevance score (highest first), and also grouped by category.

6. Overall Functioning

Indexing Workflow
- Projects are processed sequentially
- Metadata, files, people, and notes are indexed (larger files are queued for later)
- Uses a “brute-force” recursive approach to walk through project directories
  - Skips directories like node_modules, .git, .statwrap
  - Identifies eligible text files for indexing
  - Logs progress every 10 files
Document Creation Logic
- Reads file content as UTF-8 text
- Builds searchable documents with filename, content, and metadata
- Auto-generates tags based on content and file type
- Adds documents to the search index and document store
- Handles errors gracefully with debug logging
Search Functionality
- Uses field-weighted search
- Enriches results with document metadata
- Supports filtering by type or project
- Groups results by category (files, projects, people, etc.)
- Implements caching for improved performance
- Search statistics are generated to monitor performance

Challenges and End-Term Goals

In-memory Indexing Metadata Storing
Most JavaScript search libraries (like Fuse.js, Lunr, MiniSearch) store indexes entirely in memory, which can become problematic for large-scale datasets. A key challenge is designing a scalable solution that allows for disk persistence or lazy loading to prevent memory overflows.
Deciding the Weights Accordingly
An important challenge is tuning the relevance scoring by assigning appropriate weights to different aspects of the search, such as exact word matches, prefix matches, and typo tolerance. For instance, we prefer exact matches to be ranked higher than fuzzy or partial matches.
Implementing the Selected Library
Once a library is selected (based on speed, features, and compatibility with Electron + React), the next challenge is integrating it into StatWrap efficiently—ensuring local indexing, accurate search results, and smooth performance even with large projects.
Classifying Active and Past Projects in the User Interface
To improve navigation and search scoping, we plan to introduce three project sections in the interface: Pinned, Active, and Past projects. This classification will help users prioritize relevant content while enabling smarter indexing strategies.

Stay tuned for the next blog!

StatWrap: Cross-Project Searching and Classification using Local Indexing

Sat, 14 Jun 2025 00:00:00 +0000

Hello👋! I am Debangi Ghosh, currently pursuing a degree in Mathematics and Computing at IIT (BHU) Varanasi, India. This summer, I will be working on the StatWrap: Cross-Project Searching and Classification using Local Indexing project under the mentorship of Luke Rasmussen. You can view my project proposal for more details.

My project aims to address the challenges in project navigation and discoverability by integrating a robust full-text search capability within the user interface. Instead of relying on basic keyword-based search—where remembering exact terms can be difficult—we plan to implement a natural language-based full-text search. This approach involves two main stages: indexing, which functions like creating a searchable map of the content, and searching, which retrieves relevant information from that map. We will evaluate and compare available open-source libraries to choose and implement the most effective one. In addition, my project aims to enhance project organization by introducing a new classification system that clearly distinguishes between “Active” and “Past” projects in the user interface. This will improve clarity, reduce clutter, and provide a more streamlined experience as the number of projects grows.

Stay tuned for updates on my progress in the coming weeks! 🚀