Crawl4AI — веб-скрейпинг для языковых моделей

★ 73.3K

Crawl4AI is an open-source crawler and scraper built so its output can go straight into a language model. Reach for it when you need to build a knowledge base from documentation or a blog, pull articles and product pages for analysis, prepare a corpus for a RAG pipeline, or watch competitor pages for changes. The key difference from an ordinary scraper: a page comes back not as raw HTML but as clean Markdown — headings, tables, code blocks and links preserved, navigation and ads stripped. That text can go into a model or a vector database with no further cleanup. It can extract structured fields against a schema, follow links to crawl a site in depth, handle dynamic pages through a real browser, and run many jobs in parallel. Install it as a Python library or run it as a service in a container.