Back in 2017, in Nantes (France), working from an ecommerce warehouse, we benchmarked plenty of scraping solutions. Especially Scrapy, that we unsucessfuly tried to scale, before realizing that 1) it would be unmaintainable and 2) extremely costly.
To handle the thousands of simultaneous connections needed, we used Kafka, ZooKeeper, HBase; we developed a custom "Work Balancer" to handle the work-load distribution, and our own crawler, super light and blazing fast.
Since high school back in 2008, to Indie Developers, they kept working together. François is a specialist in parallel computing and is in charge of our infrastructure while Vincent focuses on frontend engineering and the Product.