Sr. Data Engineer Lead

Sr. Data Engineer Lead

23 Nov 2024
New York, New york city, 10001 New york city USA

Sr. Data Engineer Lead

Vacancy expired!

Combining consumer surveys, retail point-of-sale data, and cutting-edge analytics, The NPD Group delivers trusted retail industry research to the world’s most successful brands and retailers. We also offer careers that reward innovation and provide growth opportunities for forward-thinking problem-solvers who join our team. Speed your career growth, while working with the best and brightest in the market research industry.

Position Overview
Headquartered in Port Washington, NY with offices around the globe, The NPD Group is a leader in the collection, processing, and delivery of point-of-sale and consumer tracking data in more than 20 industries. We are expanding and seeking a senior data engineering lead who has hands on experience working with cloud and Big data platforms. You will be responsible for data engineering, data analysis and user support primarily focused on Azure Databricks and Cloudera Hadoop. You will have strong hands-on technical skills as well as being able to mentor and train other Engineers with programming languages such as Python and Scala, including tuning complex big data spark applications.

Responsibilities
  • Design, develop, and maintain highly scalable cloud-based data processing applications
  • Manage and mentor a team of 3-5 data engineers
  • Write efficient, reusable and well documented code
  • Deliver highly complex big data projects using Databricks, Data Factory and other Azure services
  • Maintain and tune existing Hadoop/Hbase applications and migrate to Azure cloud
  • Actively participate in daily agile / scrum meetings

Qualifications:
  • 7+ years of software development and design
  • 2+ years of leading a team of 3-5 software developers
  • Experience in developing and tuning spark applications using Databricks
  • Excellent understanding of spark architecture, spark SQL, data frames and tuning big data spark applications
  • Excellent with Python, Scala, SQL and shell scripting
  • Hands-on experience with cloud platforms like Azure and AWS
  • Experience in Database Design and Data Modelling
  • Excellent problem solving and analytical skills
  • Strong diagramming skills – flowcharts, data flows, etc.
  • Solid written and verbal communication skills
  • Experience with Synapse, Snowflake or Google BigQuery is a plus
  • Bachelor's degree in Computer Science or related field

Related jobs

  • Description We are hiring a Master Data Management Engineer who will work with teams in a fast-paced technology organization. As a Master Data Manatement Engineer, you will support the analytic, technical and operational aspects of enterprise data management and governance. You will support data analysis and mining activities and act as a liaison to Information Technology teams, as well as providing ongoing support for data initiatives through master data management, data quality, data lineage, and cataloging.

  • The Solutions Cloud Engineer is responsible for architecting and developing cloud-native platforms on Azure - AWS. Virtual machines, containers, serverless, network, storage, security, platform services such as analytics, database, devops pipeline, etc. Architect Integration Platform as a Service such as MuleSoft/Boomi. Developing software defined strategies for infrastructure management, application deployment, and services. Moving on-premises production and testing environments to the cloud and providing continuous technical support to both internal and external customers through the move process and afterwards. Daily management and maintenance of cloud-based infrastructure.

  • Minimum qualifications:

  • About Us

  • Lead Software Engineer, QBE Insurance Corporation, New York, NY:Responsible for working with business and IT disciplines to develop and enhance business systems that align with strategic direction and roadmap. Act as a technical subject matter expert for Majesco Policy Admin, Billing, and Digital 1st products. Oversee system changes for sound design and quality of delivery. Deliver features and functions to businesses under the project and production support model. Work with the Principal technologists on upgrading the IT infrastructure to the latest versions so the QBE has the latest and greatest technology infrastructure. Work with the demand and supply to develop work and resource forecasts for the applications delivery support. Manage the continuous improvement of development processes and methodologies. Assess and address customer impacts in the design and delivery of solutions. Review the application design of other senior members of the team and provide feedback. Perform quality reviews and lead the end-to-end implementation of a solution. Complete impact analysis and ensure quality checks are performed. Perform implementation, handover, and support activities as required. Define the best practices in the designated technology area. Estimate effort for activities, report on progress, and manage all activities within an agreed budget. Provide performance management, mentoring, and staff development.Full time, Monday to Friday, 40 hours per week, $182,054.00 per year.MINIMUM REQUIREMENTS:Bachelor’s degree in Computer Science or a related field, and 8 years of progressive, post-baccalaureate related work experience in the IT industry. In lieu of a Bachelor’s degree, employer will also accept additional 2 years of related work experience in the IT industry.Of the required experience, must have 5 years of experience in each of the following:· Majesco Policy Admin;· Billing;· Requirements analysis, solution design and development;· Working with the business and IT to develop plans and strategies in alignment with the strategic direction;· CI/CD;· SDLC and associated methodologies, including DevOps, Agile, Iterative, and Waterfall; and· Assessing customer requests impact to existing design and delivering future proof solutions.Of the required experience, must have 3 years of experience in Digital 1st products solution design and development. Telecommuting permitted up to 2 days/week.Domestic travel required up to 5%.Method to apply: Email CV to Jonathan.Stalder@us.qbe.com with Job Code KBGFJG185656-5 in the subject line

  • This Opportunity

  • Description

Job Details

  • ID
    JC23308474
  • State
  • City
  • Job type
    Permanent
  • Salary
    Depends on Experience
  • Hiring Company
    The NPD Group
  • Date
    2021-10-29
  • Deadline
    2021-12-27
  • Category

Jocancy Online Job Portal by jobSearchi.