Ideal Candidate

We’re interested in someone comfortable with a generalist and devops role. You should be knowledgeable of standard system administration tasks and have a firm understanding of the role of load balancers and cluster architecture. It’s 100x harder to write code if you don’t know how the underlying operating system works.

We are also looking for someone with a legitimate passion for technology, big data, and analyzing vast amounts of content.

This position can be located anywhere in the world with ideally at least a 4 houroverlap with the Pacific Standard Time Zone (PST / UTC-8). We’re based out of San Francisco but are migrating to the international level. If you don’t have a natural time overlap with UTC-8 you should be willing to work evenings to be able to communicate easily with the rest of the team.

Culturally, we’re a remote company and want to embrace it as a way to reward our employees.We are fine with you working in remote locations as long as you’re generally available for communication and are productive.

We want someone to come in full time as a contractor role. I suspect we will need about 40 hours from you per week.

Job Responsibilities:

• Understanding our crawler infrastructure and ensuring top quality metadata for our customers. There’s a significant batch job component to analyze the output from the crawl to ensure top quality data.

• Making sure our infrastructure is fast, reliable, fault tolerant, etc. At times this may involve diving into the source of tools like ActiveMQ, Cassandra and understand how the internals work. We contribute a LOT to Open Source development if our changes need to be given back to the community.

• Building out new products and technology that will directly interface with customers. This includes cool features like full text search, analytics, etc. It’s extremely rewarding to build something from ground up and push it to customers directly.

Architecture:

Our infrastructure consists of Java on Linux (Debian/Ubuntu) with the stack running on ActiveMQ, Cassandra, Zookeeper, and Jetty. We use Ansible to manage our boxes. We have a full-text search engine based on Elasticsearch and store our firehose API data within Cassandra.

We have a totally new stack and infrastructure at this point. We recently did a full-stack rewrite and moved all the old code to our new infrastructure. This means we have very little legacy cruft to deal with.

Here’s all the cool stuff you get to play with:

• Large Linux / Ubuntu cluster running with the OS versioned using both Ansible and our own debian packages for software distribution.

• Massive amount of data indexed from the web and social media. We index from 5-20TB of data per month and want to expand to 100TB of data per month.

• Large Cassandra install on SSD.

• SOLR / Elastic Search migration / install. We’re experimenting with bringing this up now so it would be valuable to get your feedback.

Compensation:

• Salary based on experience. We’re willing to be competitive and a great company to work for.

• Ability to work remotely at home. Live work balance is a must.