ClearPath — AI Document Intelligence Platform
Full-Stack Engineer & AI Pipeline Architect
TL;DR
Architected and built ClearPath, an AI document intelligence platform designed to decode complex official documents. Engineered a highly reliable 3-tier backend featuring a transactional outbox, BullMQ background processing, real-time SSE updates, and a 5-stage LLM pipeline grounded in official .gov/.edu sources to prevent hallucinations.
System Architecture Pipeline
↳ Upload / SSE Stream
↳ Confirm Extraction
↳ Save State & Outbox
↳ Push Real-time UI
↳ Enqueue Jobs
↳ Awaiting Verification
↳ Publish Events
↳ 5-Stage Inference
↳ Source Grounding
↳ Persist Results
↳ Publish Events
↳ LISTEN / NOTIFY
↳ Consume Preprocessing
↳ Consume AI Pipeline
↳ Subscribe Events
Tech Stack
Problem
High-stakes documents like immigration notices or benefit letters are notoriously difficult to understand, especially for non-native speakers. A simple misunderstanding can lead to missed deadlines or legal peril. Standard LLM wrappers are too risky for this because they hallucinate facts and invent URLs. A reliable, grounded, and step-by-step pipeline was required.
Constraints
Zero tolerance for lost processing jobs if a server crashes, strict requirement for LLM accuracy/grounding, protection against malicious prompt injections inside user-uploaded PDFs, and the necessity to keep the client UI updated in real-time during a long-running multi-minute analysis process.
My Role
Full-Stack Engineer & AI Pipeline Architect — designed and implemented the monorepo, database schema, custom JWT auth, BullMQ worker orchestration, SSE streaming layer, and the 5-stage AI pipeline.
Architecture
ClearPath utilizes an event-driven, decoupled architecture. The API Server handles HTTP requests and SSE connections. State changes (like a new upload) are written to a PostgreSQL outbox. The Dispatcher process listens to these outbox inserts via Postgres NOTIFY and pushes them to Redis/BullMQ. Finally, Worker processes pull from BullMQ to execute heavy preprocessing and LLM tasks, publishing their progress back to Redis Pub/Sub (for active SSE clients) and Postgres (for historical log replay).
Approach
- Prioritize reliability: Use the outbox pattern so the database transaction and queue dispatch are fundamentally tied together, preventing lost jobs
- Human-in-the-loop: Stop the pipeline after raw text extraction so the user can verify dates, contacts, and text before spending LLM tokens on analysis
- Defensive LLM engineering: Every pipeline stage utilizes strict Zod schema validation with safe fallbacks, ensuring the pipeline gracefully degrades rather than crashing
- Trust but verify: Force the LLM to verify extracted actions against real-time Tavily searches restricted to .gov and .edu domains
Responsibilities
- Designed the database schema utilizing Supabase PostgreSQL, including JSONB storage, event logs, and transactional outbox tables
- Implemented a reliable Outbox Dispatcher using PostgreSQL LISTEN/NOTIFY alongside a polling fallback to guarantee BullMQ job delivery
- Built the preprocessing worker that handles document extraction, noise removal, language detection, and pausing at the human verification gate
- Constructed the 5-stage AI worker pipeline: Document Understanding, Candidate Extraction, Grounding & Verification (via Tavily), Synthesis, and Safety Review
- Engineered the SSE (Server-Sent Events) infrastructure with Redis Pub/Sub for live updates and Postgres event replay to ensure clients never miss an event upon reconnecting
- Developed the Next.js frontend with SWR caching, drag-and-drop file uploads, real-time timeline feeds, and interactive extraction verification panels
Technical Solution
- Backend: Express 4, Node.js, TypeScript, raw pg connection pool, Custom JWT Auth (argon2 + httpOnly cookies)
- Queue & Workers: BullMQ backed by Redis (ioredis) processing distinct queue jobs across isolated Node processes
- AI & Processing: Groq API (llama-3.3-70b-versatile) for rapid inference, Tavily for grounding, custom fallback logic on schema validation failures
- Real-time: @microsoft/fetch-event-source on the client communicating with custom SSE endpoints utilizing heartbeat intervals and Last-Event-ID restoration
- Frontend: Next.js 14 App Router, Tailwind CSS, SWR for caching, Framer Motion for timeline animations
Outcome
Achieved a highly resilient pipeline that survives process restarts and network drops without losing user documents or analysis states, successfully bridging the gap between raw, intimidating government notices and actionable, human-readable checklists.
Proof Points
- Implemented deterministic transactional outbox pattern via Postgres LISTEN/NOTIFY.
- Built a pipeline that survives LLM schema failures by utilizing graceful Zod fallbacks.
- Eliminated dead-ends during reconnects by building a stateful SSE replay mechanism.
Lessons Learned
- LLMs will fail validation eventually. Building pipeline stages with safe fallbacks (e.g., returning 'needs_human_review' instead of crashing) is critical for production AI.
- The Outbox pattern is essential when combining Postgres with Redis queues. Direct enqueuing from API routes leads to race conditions and lost jobs.
- Real-time UX requires stateful backends. Relying strictly on live Pub/Sub means a 2-second client network blip loses pipeline events; storing events in Postgres and replaying via Last-Event-ID solves this.
More Screenshots
