Senior Big Data Engineer

Senior Big Data Engineer

Vacancy expired!

We are looking for a Senior Big Data Engineer/Senior Spark Platform that can take the lead on a critical new project to design and build a big data platform, which will be a key component for all our data pipelines, running daily with petabytes of data under high performance requirements, utilizing bleeding-edge technologies. Our customer is one of the world's largest technology companies based in Silicon Valley with operations all over the world. The ideal candidate is a driven, enthusiastic and technology-proficient Spark Software/Big Data Engineer, who is eager and able to develop projects from scratch interconnecting up to +15 teams and several hundreds of engineers.

Responsibilities:

• Code, develop and design a data platform to support big data pipeline and infrastructure for 100s of engineers
  
• Drive development end-to-end for specific components
  
• Contribute to project discussions, collaborate directly with lead architects and present results to key stakeholders
  
• Write detailed design documentation, present decisions and motivate these
  
• Design, build, support and continuously enhance the project code base
  
• Work inside a team of industry experts on the cutting edge Big Data technologies to develop solutions for deployment at massive scale
  

Requirements:

• +8 years experience working with Big Data - Spark and Hadoop for a minimum of 4 years
  
• +3 years of experience working with event-messaging systems - Kafka in particular
  
• Strong knowledge of Scala and coding skills - alternatively very strong Java with 1-2 year of Scala
  
• Strong understanding of the challenges of building end-to-end big data pipelines for a large variety of use-cases at scale
  
• Strong understanding of the challenges of synching many disjunct big data technologies
  
• Strong communication skills
  
• Worked with/understanding of streaming processes and technologies - Cassandra, Spark Streaming
  
• Worked with big data pipelines at terabyte/petabyte scale
  
• Worked with HDFS
  

What will be a big plus:

• Understanding how to run Spark on Kubernetes
  
• Experience working with scheduling technologies - Airflow, Azkaban
  
• Experience with JVM build systems (SBT, Maven, Gradle)
  

We offer:

• Opportunity to work on bleeding-edge projects
• Work with a highly motivated and dedicated team
• Competitive salary
• Flexible schedule
• Medical insurance
• Benefits program
• Corporate social events

NB:
Placement and Staffing Agencies need not apply. We do not work with C2C at this time.
At this moment, we are not able to process H1B transfers.

About us:
Grid Dynamics is the engineering services company known for transformative, mission-critical cloud solutions for retail, finance and technology sectors. We architected some of the busiest e-commerce services on the Internet and have never had an outage during the peak season. Founded in 2006 and headquartered in San Ramon, California with offices throughout the US and Eastern Europe, we focus on big data analytics, scalable omnichannel services, DevOps, and cloud enablement.