Back to All Projects
AI & Machine LearningLiveStatus: Completed

High-Throughput RAG & Semantic Search Gateway

Sub-50ms enterprise retrieval pipeline combining dense vector embeddings, BM25 sparse search, cross-encoder reranking, and semantic Redis caching.

My RoleAI & Backend Infrastructure Engineer
Duration3 Months
Team SizeSolo / Core Developer
DeploymentLive
High-Throughput RAG & Semantic Search Gateway

The Problem

Standard vector-only search systems suffered from low keyword precision on technical acronyms and domain-specific terms. Furthermore, un-cached embedding inference created latency spikes exceeding 800ms per query under heavy load.

The Engineered Solution

Built a hybrid retrieval architecture combining dense vector embeddings (using BAAI/bge-large) with BM25 sparse inverted indices via reciprocal rank fusion (RRF). Added an in-memory Redis semantic cache that matches semantically equivalent queries above a cosine threshold of 0.94 without invoking downstream models.

System Design

Technical Architecture

System Topology & Specifications

Hybrid Dense-Sparse RAG Architecture

Two-stage retrieval pipeline with semantic caching and reciprocal rank fusion.

Hybrid Dense-Sparse RAG Architecture

Data Flow & Execution Pipeline

Step 01 of 07
01
1. Incoming user query normalized and hashed.
02
2. Redis semantic cache queried for cosine similarity > 0.94.
03
3. If cache hit, return cached result immediately (<12ms).
04
4. If miss, generate dense embedding and compute BM25 sparse vectors in parallel.
05
5. Reciprocal Rank Fusion combines both candidate pools into top 25 documents.
06
6. ONNX cross-encoder reranks top 25 into final top 5 with relevance scores.
07
7. Final context returned and cached in Redis.

Component Architecture & Responsibilities

Semantic Cache (Redis)Sub-15ms Ingress Cache

Caches query-result pairs with vector index for fast semantic lookup.

Hybrid Search Core (Qdrant & BM25)Dual-Engine Retrieval

Performs simultaneous high-dimensional vector search and BM25 inverted index lookups.

Reranker (ONNX Runtime)Context Optimization

Quantized cross-encoder scoring pairs to eliminate false positive retrieval.

Deep Dive

Engineering Challenges & Resolution

Every serious engineering effort confronts unexpected technical friction. Here is how I approached and resolved the most complex roadblocks.

Why It Was Difficult

Running full cross-encoder reranking on top 100 candidate documents added 350ms of CPU compute per query.

Engineering Approach

Implemented a two-stage filter: dense+sparse RRF narrows candidates to top 25, followed by a quantized, lightweight MiniLM cross-encoder running in batch ONNX runtime.

Solution & Outcome

Reduced reranking overhead from 350ms to 24ms while maintaining 98.2% of full cross-encoder NDCG@10 precision.

Accountability

My Direct Contributions

  • Designed and benchmarked the Reciprocal Rank Fusion (RRF) algorithm combining BM25 and dense embeddings.
  • Integrated Redis semantic caching with cosine similarity thresholding, reducing embedding latency to <12ms for repeated semantics.
  • Optimized document ingestion pipelines using asynchronous chunking and batch vectorization.
  • Configured Prometheus custom metrics for search latency, cache hit ratios, and reranker execution times.
Verification & Business Impact

Measurable Results & Outcomes

P95 End-to-End Latency

P95 Query Latency

38ms

Achieved P95 retrieval latency across 500,000 document index.

Semantic Cache Hit Ratio

Semantic Cache Hit Rate

41.8%

Queries served directly from cache without LLM/Embedding invocation.

Technology Stack & Tooling

PythonFastAPIPyTorchQdrantRedisDockerPrometheusGrafanaAWS S3

Explore Related Systems

View all
Autonomous Enterprise Multi-Agent Orchestration Platform
Generative AILive

Autonomous Enterprise Multi-Agent Orchestration Platform

Production-grade multi-agent engine coordinating distributed research, code generation, and verification workflows with human-in-the-loop checkpoints.

Impact Highlight
Task Completion Success Rate
92.4%
PythonLangGraphFastAPIQdrantNext.js+6
GitOps Cloud-Native Kubernetes Infrastructure Platform
DevOps & CloudPrivate Deployment

GitOps Cloud-Native Kubernetes Infrastructure Platform

Automated AWS EKS infrastructure deployed via Terraform IaC, ArgoCD continuous delivery, Istio service mesh, and Vault secret management.

Impact Highlight
Deployment Frequency & Velocity
10x
KubernetesAWS EKSTerraformArgoCDDocker+5