WaterCrawl — сбор и парсинг данных с сайтов

★ 2K

WaterCrawl is an open-source crawling and data-extraction tool you host yourself. Reach for it when web content must be collected regularly and under control: populate a knowledge base, pull catalogues and articles, prepare data for models. Crawl depth and speed are configurable, with several levels of extraction detail from quick to deep. Built on Python, Django, Scrapy and Celery, it copes with large volumes and runs on a schedule. The difference from a plain scraper is that this is a finished service with a job queue and an interface, not a library you must wrap yourself. As with any data collection, respect site terms of use and data-protection law — that responsibility is the user's.