Cluster Interfaces is responsible for all of the entry points into Quantcast’s big data services. We develop a dashboard through which data pipelines are scheduled and executed. The dashboard provides a holistic view into the company’s data pipelines, including their dependency trees, capacity needs, and execution times. It also allows users to define data retention policies, monitoring, and alerting. The dashboard is similar to the open source project Azkaban.

The team also develops and operates a SQL-On-MapReduce platform along with a data catalog. Quantcast’s SQL-On-MapReduce platform is very similar to Hive, providing company wide easy access to our core data assets.

Cluster Interfaces is looking for an individual to drive improvements to the systems used to schedule, debug and analyze map/reduce pipelines while facilitating the management of shared cluster resources and capacity planning. Modernize the front-ends of these internal web-services using state of the art javascript frameworks, implement testing best practices, and develop rich service oriented APIs.

Responsibilities

• Consolidate the interfaces to Quantcast’s massive compute cluster

• Design rich data-visualizations to aid with analysis and resource management tasks

• Mentor and grow team members

• Work to understand the needs of customers in order to serve them better

• Contribute to the technical roadmap of the team