This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
https://github.com/istresearch/scrapy-cluster
dev