High-Throughput RAG & Semantic Search Gateway
Sub-50ms enterprise retrieval pipeline combining dense vector embeddings, BM25 sparse search, cross-encoder reranking, and semantic Redis caching.
The Problem
Standard vector-only search systems suffered from low keyword precision on technical acronyms and domain-specific terms. Furthermore, un-cached embedding inference created latency spikes exceeding 800ms per query under heavy load.
The Engineered Solution
Built a hybrid retrieval architecture combining dense vector embeddings (using BAAI/bge-large) with BM25 sparse inverted indices via reciprocal rank fusion (RRF). Added an in-memory Redis semantic cache that matches semantically equivalent queries above a cosine threshold of 0.94 without invoking downstream models.
Technical Architecture
Hybrid Dense-Sparse RAG Architecture
Two-stage retrieval pipeline with semantic caching and reciprocal rank fusion.
Data Flow & Execution Pipeline
Step 01 of 07Component Architecture & Responsibilities
Caches query-result pairs with vector index for fast semantic lookup.
Performs simultaneous high-dimensional vector search and BM25 inverted index lookups.
Quantized cross-encoder scoring pairs to eliminate false positive retrieval.
Engineering Challenges & Resolution
Every serious engineering effort confronts unexpected technical friction. Here is how I approached and resolved the most complex roadblocks.
Running full cross-encoder reranking on top 100 candidate documents added 350ms of CPU compute per query.
Implemented a two-stage filter: dense+sparse RRF narrows candidates to top 25, followed by a quantized, lightweight MiniLM cross-encoder running in batch ONNX runtime.
Reduced reranking overhead from 350ms to 24ms while maintaining 98.2% of full cross-encoder NDCG@10 precision.
My Direct Contributions
- Designed and benchmarked the Reciprocal Rank Fusion (RRF) algorithm combining BM25 and dense embeddings.
- Integrated Redis semantic caching with cosine similarity thresholding, reducing embedding latency to <12ms for repeated semantics.
- Optimized document ingestion pipelines using asynchronous chunking and batch vectorization.
- Configured Prometheus custom metrics for search latency, cache hit ratios, and reranker execution times.
Measurable Results & Outcomes
P95 Query Latency
Achieved P95 retrieval latency across 500,000 document index.
Semantic Cache Hit Rate
Queries served directly from cache without LLM/Embedding invocation.
Technology Stack & Tooling
Explore Related Systems
View allAutonomous Enterprise Multi-Agent Orchestration Platform
Production-grade multi-agent engine coordinating distributed research, code generation, and verification workflows with human-in-the-loop checkpoints.
GitOps Cloud-Native Kubernetes Infrastructure Platform
Automated AWS EKS infrastructure deployed via Terraform IaC, ArgoCD continuous delivery, Istio service mesh, and Vault secret management.