A client of ours in Seattle WA is looking for a Sr. Data Pipeline Engineer for a Fulltime opportunity.
Skillset:
AURORA
SPARK
AIRFLOW
DOCKER
AWS
GENERAL SUMMARY:
Our combined Disney and Hulu team is looking for hardworking team players to join the Ad
Engineering team, who will thrive upon the legacy of Disney and Hulu, embrace unconventional
thinking, and who are passionate about contributing to The Walt Disney Company's direct-to-consumer (DTC) experience through strategic hard work and determination.
We are looking for a Senior Data Pipeline Engineer with experience in building data pipeline
solutions integrating components built on top of an AWS technology stack. If you have
experience building financial applications, take ownership in the technical direction of your
team's product we'd like to talk with you.
The Data Pipeline Engineer will be responsible for building efficient data pipelines that populate
our data lake, apply calculations and aggregations across the data set and load the results into
SQL databases that serve both analytical and operational use cases. This role will be working
closely with different engineering teams and product managers to meet data requirements of
various initiatives in Ad Engineering.
What you will do:
Think and communicate critically about architecture, design, and best practices and guide your team to adopting them.
Design data systems that allow managed growth of the data model to minimize risk and cost of change.
Write transformation and validation code that applies complex data aggregation and calculation using SQL and Python
Drive implementation of automated testing for data pipelines within a CI environment
Create new pipelines or rewrite existing pipelines and build reusable components at scale to support accounting functions, as well as reporting & analytics.
Collaborate with other Disney and Hulu teams to identify and document shifting data requirements while also advocating for a minimal change set for your team.
Solve complex data issues and perform root cause analysis to proactively resolve product and operational issues.
Collaborate with leadership and other engineers to develop technical story backlog derived from high level business requirements and design collaboration and estimating story points.
BS or MS in Computer Science, a related field, or equivalent industry experience
3 years of professional experience engineering complex, high-volume data pipelines using SQL, Python, and Airflow
3 years of experience building cloud scalable and high-performance data lake / data warehouse solutions using AWS products - S3, Athena, Glue, and EMR
Experience with binary data serialization formats such as Parquet
Deep understanding of data structures and algorithms
Understanding of code versioning tools such as GIT
Have a passion for data solutions
Preferred Qualifications:Nice to Have:
Exposure to AWS cloud data pipeline tools such as Managed Airflow and Glue
Experience integrating with Ad Tech platforms such as Operative and STAQ
Exposure and opinions regarding alternate orchestration tooling beyond Airflow
Understanding of SOX compliance needs and how they affect system design.
Have worked with a variety of Airflow Operator types, including REST, Lambda, ECS
Can flex between Python and Javascript/Typescript.
Technical Environment:
Aurora/Hive (databases)
Spark (large-scale data processing)
Airflow (workflow management)
Docker (software packaging and delivery)
AWS (development and hosting)
Required Education:
BS/MS in Computer Science or sim. field
Additional Information:
This team is considered Hybrid so looking for someone that's close to a Hulu Office. So.Cal/Seattle/NYC - Can be remote to start but expectation is in office part time if needed.
Job Type:Contract Location: Preferred location is Seattle WA