Sr. Data Pipeline Engineer

Sr. Data Pipeline Engineer

15 Aug 2024
Washington, Seattle-tacoma, 98141 Seattle-tacoma USA

Sr. Data Pipeline Engineer

Vacancy expired!

Job Summary:

  • A client of ours in Seattle WA is looking for a Sr. Data Pipeline Engineer for a Fulltime opportunity.
Skillset:
  • AURORA
  • SPARK
  • AIRFLOW
  • DOCKER
  • AWS
GENERAL SUMMARY:
  • Our combined Disney and Hulu team is looking for hardworking team players to join the Ad
  • Engineering team, who will thrive upon the legacy of Disney and Hulu, embrace unconventional
  • thinking, and who are passionate about contributing to The Walt Disney Company's direct-to-consumer (DTC) experience through strategic hard work and determination.
  • We are looking for a Senior Data Pipeline Engineer with experience in building data pipeline
  • solutions integrating components built on top of an AWS technology stack. If you have
  • experience building financial applications, take ownership in the technical direction of your
  • team's product we'd like to talk with you.
  • The Data Pipeline Engineer will be responsible for building efficient data pipelines that populate
  • our data lake, apply calculations and aggregations across the data set and load the results into
  • SQL databases that serve both analytical and operational use cases. This role will be working
  • closely with different engineering teams and product managers to meet data requirements of
  • various initiatives in Ad Engineering.
What you will do:
  • Think and communicate critically about architecture, design, and best practices and guide your team to adopting them.
  • Design data systems that allow managed growth of the data model to minimize risk and cost of change.
  • Write transformation and validation code that applies complex data aggregation and calculation using SQL and Python
  • Drive implementation of automated testing for data pipelines within a CI environment
  • Create new pipelines or rewrite existing pipelines and build reusable components at scale to support accounting functions, as well as reporting & analytics.
  • Collaborate with other Disney and Hulu teams to identify and document shifting data requirements while also advocating for a minimal change set for your team.
  • Solve complex data issues and perform root cause analysis to proactively resolve product and operational issues.
  • Collaborate with leadership and other engineers to develop technical story backlog derived from high level business requirements and design collaboration and estimating story points.
  • BS or MS in Computer Science, a related field, or equivalent industry experience
  • 3 years of professional experience engineering complex, high-volume data pipelines using SQL, Python, and Airflow
  • 3 years of experience building cloud scalable and high-performance data lake / data warehouse solutions using AWS products - S3, Athena, Glue, and EMR
  • Experience with binary data serialization formats such as Parquet
  • Deep understanding of data structures and algorithms
  • Understanding of code versioning tools such as GIT
  • Have a passion for data solutions
Preferred Qualifications: Nice to Have:
  • Exposure to AWS cloud data pipeline tools such as Managed Airflow and Glue
  • Experience integrating with Ad Tech platforms such as Operative and STAQ
  • Exposure and opinions regarding alternate orchestration tooling beyond Airflow
  • Understanding of SOX compliance needs and how they affect system design.
  • Have worked with a variety of Airflow Operator types, including REST, Lambda, ECS
  • Can flex between Python and Javascript/Typescript.
Technical Environment:
  • Aurora/Hive (databases)
  • Spark (large-scale data processing)
  • Airflow (workflow management)
  • Docker (software packaging and delivery)
  • AWS (development and hosting)
Required Education:
  • BS/MS in Computer Science or sim. field
Additional Information:
  • This team is considered Hybrid so looking for someone that's close to a Hulu Office. So.Cal/Seattle/NYC - Can be remote to start but expectation is in office part time if needed.
Job Type: Contract
Location: Preferred location is Seattle WA

Job Details

Jocancy Online Job Portal by jobSearchi.