As a Big Data engineer, you need to be intimately familiar with assembling large, complex data sets to satisfy functional and technical business requirements. Analytical skills related to working with unstructured datasets optimize data delivery, automate manual processes, re-design infrastructure for improved scalability and identify, design and implement data ingestion, data delivery process improvements.
Collaborate with analytics, data scientists and product teams to assist in building and optimizing innovative products and services to market
Develop analytics tools that utilize data pipeline to provide actionable insights into customer acquisition, operational efficiency and key business performance metrics.
Build big data pipelines, data sets and conduct performance tuning
Working knowledge of message queuing, stream processing, and highly scalable ‘big data' data stores.
Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS ‘big data' technologies.
Experience supporting and working with cross-functional teams in a dynamic environment
Keep data separated and secure across architectural boundaries through multiple data centers and AWS regions.
What you know:
Hadoop, Spark, Kafka, etc. Experience with relational SQL and NoSQLdatabases such as Cassandra.
Experience with AWS cloud services: EC2, EMR, Athena
Experience with object-oriented/object function scripting languages: Python, Java, C, Scala,
Advanced SQL knowledge and experience working with relational databases, query authoring (SQL) as well as familiarity with unstructured datasets.
Deep problem-solving skills to perform root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
Build processes supporting data transformation, data structures, metadata, dependency and workload management.
Proven experience in manipulating, processing and extracting value from large disconnected datasets.
Strong project management and organizational skills
Experience supporting and working with cross-functional teams in a dynamic environment.
Experience
5+ years of experience in a Data Engineer role
Education:
Graduate degree in Computer Science, Statistics, Informatics, Information Systems or another quantitative field.
