Sr Data Engineer

Sr Data Engineer

Vacancy expired!

Ideal candidate is local and able to work a hybrid schedule. Will consider full remote for non-local candidates. No sponsorship available

MUST HAVE:

• Strong SQL experience

• Python scripting experience

• AWS cloud stack experience

Job Summary
As a DataOps engineer, you will be responsible for managing our data platform and enabling our mission of improving Customer Average Value. You will work with cross functional partners and third-party vendors to support our customer data assets by maintain the data pipelines to acquire, organize, and aggregate customer data from various sources for use in direct/email marketing, targeted media campaigns and analytics.

Responsibilities

• Provide production support, including production defect resolution within reporting, ETL, 3rd party integrations, EDW, and master data for customer platform
• Maintain BI reliability standards and documentation
• Ensure release packages are implemented with complete information and support the maintenance as part of the System Development Life Cycle.
• Responsible for all change control, system compliance, and system audit reviews related to Customer platform.
• Monitor Data Observability and ensure data integrity of the entire ecosystem include data pipelines, lakehouse and redshift environment
• Enhance, Deploy and maintain data processing pipelines using cloud technology such as AWS Airflow, Redshift, and Python.
• Enhance, Deploy, and maintain serverless data pipelines using Event Bridge, Kinesis, AWS Lambda, S3, and Glue.
• Deploy and maintain Power BI, YellowFin, and AWS Quick Sight reports and dashboards.
• Ensure high operational efficiency and quality of your solutions to meet SLAs and support commitment to our customers (Data Science, Data Analytics teams).

Skills & Requirements

• Bachelor’s degree in analytics, statistics, engineering, math, economics, computer science, information technology, or related discipline.
• 7+ years of experience working with Business Intelligence / Data Warehouse concepts and principles.
• 5+ years of experience in supporting BI Production support processes including ETL data loads.
• 5+ years of experience to support production operational processes based on ETL data loads tools, performance monitoring, security access, troubleshooting, operational issues/processes, and implementations.
• 3-5 years of experience managing large-scale, mission-critical data pipelines and features using modern big data architectures.
• 3-5 years of hands-on experience in Strong coding skills with Python/Pyspark/Spark and SQL.
• 3+ years of hands-on experience in ETL tools such as Informatica.
• Experience in ETL tools, AWS Tech Stack – Redshift, Glue, EMR, Redshift Spectrum, and Athena.
• Ability to run queries to troubleshoot, create reports and provide analysis. Demonstrated ability to analyze large data sets to identify gaps and inconsistencies, provide data insights, and advance effective solutions.
• Expert knowledge in writing complex SQL and ETL development with experience processing extremely large datasets.
• Experience developing Infrastructure as a Code by creating CloudFormation templates using YAML and JSON.
• Experience integrating data from multiple data sources and file types such as JSON, Parquet, and Avro formats.
• Experience supporting and working with cross-functional teams in a dynamic environment
• Strong quantitative and communication skills.
• Experience with CI/CD tools like Gitlab and Terraform.
• Experience with data quality processes, data quality checks, validations, data quality metrics definition, and measurement.
• Experience in data quality tools such as Informatica DQ or Talend DQ tools.