GPT Crawler — сайт в базу знаний для ИИ-ассистента

★ 22.4K

GPT Crawler is a tool that crawls a given website, gathers its content, and produces a knowledge file from which you can make your own GPT assistant about that site. Reach for it when you want to quickly turn a website or documentation into a base for an AI assistant: you give a start URL and crawl rules, and the crawler collects content into one file that you upload into a custom GPT (or your own RAG) so the bot answers from the site's material. Its focus is crawling a site and preparing a knowledge file to build a GPT/assistant about it, not a finished chatbot with a UI (firestarter/DocsGPT) or a general data scraper (Firecrawl/crawl4ai): the value is getting a corpus for an assistant about a specific site in a few steps. Use it lawfully, respecting site terms and copyright. It suits people who want to quickly make an AI assistant about their site or documentation by gathering its content into one file.