Intelligent Web Scraper & Local RAG Chatbot
Site-to-knowledge-base platform with an embeddable widget
AI · RAG · SaaS Backend
Crawling engine, RAG ingestion pipeline, LLM workflows, billing integration
Private production project
Overview
A scalable full-site crawling engine with adaptive scraping strategies for static and dynamic sites, feeding a Retrieval-Augmented Generation pipeline that powers an embeddable site chatbo[...]
The challenge
Websites are inconsistent — some render server-side, some only exist after JavaScript runs. Turning that into grounded answers means crawling adaptively, chunking and embedding reliably,[...]
Contribution
- Designed a scalable full-site crawling engine with adaptive scraping strategies for dynamic and static websites, powering an embeddable site chatbot widget.
- Built the backend ingestion pipeline with FastAPI, LangChain, FAISS and HuggingFace embeddings for Retrieval-Augmented Generation.
- Implemented locally hosted Gemma LLM workflows with persistent session storage on Supabase for document-grounded conversational querying.
- Integrated Razorpay subscription billing with two-step verification for secure access control and plan-based feature gating.
Architecture & workflow
How the system moves from input to outcome.
Core features
Full-site crawling
Adaptive strategies per site type rather than one brittle scraper.
RAG ingestion
Chunking, embedding and indexing as a repeatable pipeline.
FAISS vector search
Fast similarity retrieval over site content.
Local LLM inference
Gemma hosted locally through Ollama.
Persistent sessions
Conversation state stored in Supabase.
Subscription billing
Razorpay with two-step verification and plan-based feature access.
Embeddable widget
Chatbot designed to drop into an existing site.
Technical highlights
- Adaptive crawling across static and dynamic sources.
- Locally hosted LLM workflows instead of a third-party inference dependency.
- Commercial layer — subscriptions and feature gating — built into the product.