Scrapinghub is looking for an Erlang software developer to join our Crawlera team.

Crawlera is a smart downloader designed specifically for web crawling and scraping. It allows crawler developers to crawl quickly and reliably by managing thousands of proxies internally. It is part of the Scrapinghub platform, the world’s most comprehensive web crawling stack which powers crawls of over 4 billion pages a month.

Scrapinghub helps companies, ranging from Fortune 500 enterprises to early stage startups, turn web content into useful data with a cloud-based web crawling platform, off-the-shelf datasets, and turn-key web scraping services.

Join us in making the world a better place for web crawler developers with our team of top talented engineers working remotely from more than 30 countries.

Responsibilities

• Develop, maintain and support a high load distributed system.

• Analyze our current and historical Crawlera usage to augment and enhance its routing and rotation logic.

• Leverage the Scrapinghub platform to provide extended functionality, both to end users and for internal purposes.

• Identify and resolve performance and scalability issues with distributed crawling at scale.

• Liaison with other platform teams to provide Crawlera with the best possible integration to the growing Scrapinghub platform.