Senior Big Data Engineer

  • Location

    San Francisco

  • Sector:

    Data Science

  • Job type:


  • Salary:

    Circa $140,000- $190,000

  • Contact email:


  • Consultant:

    Jason Rumney

As a Senior or Lead Software Engineer you’ll help researching and then designing and building new solutions to identify suspect or malicious behaviour. You bring ample architecture, design, and technical leadership to bear with leading-edge big data techniques. Your expertise is key to developing highly scalable, extensible solutions to process high-volume data streams in real time or mine very large internal and external datasets. Your primary focus is architecting, designing, building, and scaling-up software for anomaly detection, behavior-based analysis, and pattern recognition, all leveraging contemporary technologies such as Apache Spark, HBase, Storm, and others. You bring exceptional communication and design skills, but also enjoy hands-on coding. 
If you’re interested in building scalable software in big data, have the breadth of skills for applied research through production-quality software development, and like working with a small, but highly visible team, this is the job for you!  

Primary Responsibilities

  • Develop scalable, extensible solution architectures leveraging contemporary big-data technologies in an evolving operational framework
  • Design and build production-quality, highly scalable, extensible solutions from POCs.
  • Participate in and help guide research POCs, including design, coding, and performance and efficacy measurement.
  • Contribute to software quality and efficiency through code reviews, test-driven development, documentation, and continuous integration.

 Required Qualifications

  • 4+ years of software development 
  • Bachelor’s degree in Computer Science or related discipline (Master’s preferred)
  • Extensive experience with Java, Object-Oriented Programming, and Python or Scala
  • Extensive experience using Salesforce or Migrating Salesforce API's
  • Demonstrated experience designing and coding in big-data components such as Hadoop and Apache Spark, HBase, and Storm
  • Excellent communication, leadership, and collaboration skills  

Preferred Qualifications

  • Experience in applying big-data techniques to security, specifically in anomaly detection, pattern recognition, behavior-based analysis, and correlation.
  • Strong understanding of security, including threat propagation and malware analysis
  • Experience developing in open-source machine-learning libraries such as Apache Mahout or MLLib
  • Experience in applied research, either building POCs or building production-quality software from POCs

To find out more please get in touch