Back to All Projects
Backend EngineeringLiveStatus: Completed

Real-Time LLM Observability & Guardrails Proxy

High-performance streaming reverse proxy enforcing prompt injection defense, PII masking, token cost allocation, and latency analytics.

My RoleFull-Stack & Systems Engineer
Duration2 Months
Team SizeSolo project
DeploymentLive
Real-Time LLM Observability & Guardrails Proxy

The Problem

Companies deploying generative AI features lacked centralized control over unpredictable API costs, had no automated defenses against indirect prompt injection, and risked leaking sensitive customer PII to external model providers.

The Engineered Solution

Developed an asynchronous proxy in FastAPI and Python with streaming token parsers. Applied regex and transformer-based entity extractors to mask PII on the fly, evaluated semantic safety classifiers before request forwarding, and logged token consumption metrics directly to PostgreSQL and Grafana.

System Design

Technical Architecture

System Topology & Specifications

Streaming Security & Observability Gateway

Non-blocking proxy architecture inspecting inbound prompts and outbound token streams.

Streaming Security & Observability Gateway

Data Flow & Execution Pipeline

Step 01 of 06
01
1. Application sends OpenAI-compatible request to Guardrails Proxy.
02
2. Rate limiter checks API key token quota against Redis sliding window.
03
3. Security scanner tests prompt for jailbreaks and scrubs PII.
04
4. Sanitized request forwarded to downstream model endpoint.
05
5. Response tokens streamed back through parser while logging token counts.
06
6. Telemetry pushed asynchronously to database and Prometheus.

Component Architecture & Responsibilities

Reverse Proxy EngineTraffic Ingress

Handles HTTP/2 and SSE streams with async event loops.

PII Masking & ScannerSafety Defense

Performs entity recognition and prompt injection heuristics.

Telemetry EngineObservability

Tracks tokens, costs, and response times per team and model.

Deep Dive

Engineering Challenges & Resolution

Every serious engineering effort confronts unexpected technical friction. Here is how I approached and resolved the most complex roadblocks.

Why It Was Difficult

Inspecting and buffering streaming LLM tokens for safety checks risked breaking the interactive feel of real-time chat applications.

Engineering Approach

Implemented a rolling window token tokenizer that verifies syntax and safety asynchronously while streaming safe chunks to the client.

Solution & Outcome

Kept added proxy latency to under 8ms for the initial token (TTFT) and under 1ms for subsequent token streaming.

Accountability

My Direct Contributions

  • Built SSE (Server-Sent Events) streaming token pass-through with sub-5ms proxy overhead.
  • Created token budgeting rate limiters per API key with Redis sliding window algorithms.
  • Integrated Hugging Face lightweight models for real-time prompt injection detection.
  • Developed a Next.js administrative dashboard for auditing flagged requests and token usage trends.
Verification & Business Impact

Measurable Results & Outcomes

PII Sanitization Accuracy

PII Leak Prevention

100%

Zero undetected SSN, credit card, or email leaks across synthetic attack suite.

Added Proxy Latency (P99)

Proxy Overhead

<8ms

Negligible latency added to downstream LLM response streaming.

Technology Stack & Tooling

PythonFastAPIRedisNext.jsDockerPostgreSQLPrometheus

Explore Related Systems

View all
Autonomous Enterprise Multi-Agent Orchestration Platform
Generative AILive

Autonomous Enterprise Multi-Agent Orchestration Platform

Production-grade multi-agent engine coordinating distributed research, code generation, and verification workflows with human-in-the-loop checkpoints.

Impact Highlight
Task Completion Success Rate
92.4%
PythonLangGraphFastAPIQdrantNext.js+6
High-Throughput RAG & Semantic Search Gateway
AI & Machine LearningLive

High-Throughput RAG & Semantic Search Gateway

Sub-50ms enterprise retrieval pipeline combining dense vector embeddings, BM25 sparse search, cross-encoder reranking, and semantic Redis caching.

Impact Highlight
P95 Query Latency
38ms
PythonFastAPIPyTorchQdrantRedis+4