DocIngest Suite

DocIngest Documentation Suite LIVE EDGE SUITE

Documentation Crawler & Corpus Viewer

DocIngest is a high-performance, self-hosted documentation crawler, markdown converter, and semantic search engine. It crawls canonical developer documentation libraries and structures them for immediate ingestion into Model Context Protocol (MCP) servers and AI reasoning loops.

Index New Docs (Add) 🚀 Browse Searchable Corpus (View) 📚

---

◈ Interactive Modules

• DocIngest URL Ingestion (/homelab-projects/docingest/add or /add)
  Submit target documentation URLs with inclusion and exclusion path filters (/docs/*, */v1/*) to crawl, convert to clean Markdown, and persist to the PostgreSQL database.

• DocIngest Corpus Viewer (/homelab-projects/docingest/view or /view)
  Full-text search indexed documentation, preview full Markdown in an elevated modal, copy context directly to clipboard, or download Markdown files.

---

◈ Architecture & Technical Stack

The DocIngest engine operates as a decoupled microservices architecture designed for zero-trust resilience and high-throughput document extraction:

• Ingestion & Crawling: Firecrawl microservice orchestrating headless Playwright instances to render dynamic SPAs, strip navigation boilerplate, and extract pure markdown.
• Queue & Event Bus: Redis message queue for job scheduling, status polling, and concurrency control.
• Storage Tier: PostgreSQL database with JSONB schema for indexing document hierarchies and fast metadata retrieval.
• MCP Integration: Exposes indexed corpora directly to local agent swarms through the Model Context Protocol (find-docs, read-docs, query-docs).