Real-Time LLM Observability & Guardrails Proxy
High-performance streaming reverse proxy enforcing prompt injection defense, PII masking, token cost allocation, and latency analytics.
The Problem
Companies deploying generative AI features lacked centralized control over unpredictable API costs, had no automated defenses against indirect prompt injection, and risked leaking sensitive customer PII to external model providers.
The Engineered Solution
Developed an asynchronous proxy in FastAPI and Python with streaming token parsers. Applied regex and transformer-based entity extractors to mask PII on the fly, evaluated semantic safety classifiers before request forwarding, and logged token consumption metrics directly to PostgreSQL and Grafana.
Technical Architecture
Streaming Security & Observability Gateway
Non-blocking proxy architecture inspecting inbound prompts and outbound token streams.
Data Flow & Execution Pipeline
Step 01 of 06Component Architecture & Responsibilities
Handles HTTP/2 and SSE streams with async event loops.
Performs entity recognition and prompt injection heuristics.
Tracks tokens, costs, and response times per team and model.
Engineering Challenges & Resolution
Every serious engineering effort confronts unexpected technical friction. Here is how I approached and resolved the most complex roadblocks.
Inspecting and buffering streaming LLM tokens for safety checks risked breaking the interactive feel of real-time chat applications.
Implemented a rolling window token tokenizer that verifies syntax and safety asynchronously while streaming safe chunks to the client.
Kept added proxy latency to under 8ms for the initial token (TTFT) and under 1ms for subsequent token streaming.
My Direct Contributions
- Built SSE (Server-Sent Events) streaming token pass-through with sub-5ms proxy overhead.
- Created token budgeting rate limiters per API key with Redis sliding window algorithms.
- Integrated Hugging Face lightweight models for real-time prompt injection detection.
- Developed a Next.js administrative dashboard for auditing flagged requests and token usage trends.
Measurable Results & Outcomes
PII Leak Prevention
Zero undetected SSN, credit card, or email leaks across synthetic attack suite.
Proxy Overhead
Negligible latency added to downstream LLM response streaming.
Technology Stack & Tooling
Explore Related Systems
View allAutonomous Enterprise Multi-Agent Orchestration Platform
Production-grade multi-agent engine coordinating distributed research, code generation, and verification workflows with human-in-the-loop checkpoints.
High-Throughput RAG & Semantic Search Gateway
Sub-50ms enterprise retrieval pipeline combining dense vector embeddings, BM25 sparse search, cross-encoder reranking, and semantic Redis caching.