Data Engineer with Spark

Data Engineer with Spark

31 Jan 2024
Michigan, Livonia, 48150 Livonia USA

Data Engineer with Spark

Vacancy expired!


Data Engineer with Spark exp (Please share the 10 yrs exp profiles)
Client location : Livonia,MI
Duration:6+ Months
Responsibilities Work with data engineering team to define and develop data ingestion, validation, transformation and data engineering code.
Develop open source platform components using Spark, Scala, Java, Oozie, Hive and other components
Document code artifacts and participate in developing user documentation and run books
Troubleshoot deployment to various environments and provide test support.
Participate in design sessions, demos and prototype sessions, testing and training workshops with business users and other IT associates Qualifications: Data Engineers are required to design and build data products and data pipelines. They will ensure the robust flow of data from acquisition through curation, and governance. Data Engineers will enable data as a service and drive all critical data driven initiatives.
Experts of the data and its application by users
Understands data landscape and environments: sources, elements, update freq, completeness, stewards/contacts, platforms
Manages ETL: use programming and tools for data ingestion, configure pipelines, apply transformations and decoding, integrate and fuse data, move and securely deliver
Translates business requirements to build repeatable, sustainable, efficient, coded processes that can be productionized by Software Engineers and readily modified by other Data Engineers
Creates POC processes in Dev/QA and works with Software Engineers to productionize and align best coding and process practices that maximize efficiency, speed, stability, system resources and capabilities
Leverages frameworks in place with big data tools: Hadoop, Spark, Python, Kafka, etc.
Experience with relational SQL and NoSQL databases
Awareness of and compliance with: data privacy, security, legal and contractual guidelines
Incorporates data quality and privacy checks/alerts to minimize bad data being consumed by end users, models and dashboards, and to protect customer data; revises checks for new data issues
Maintains feedback loop with Data Stewards on data issues, standards, fit for use (Data Stewardship is a subset of data engineering which would include responsibilities like data curation)
Validates data products and pipelines are functioning as expected following system or application upgrades, source changes, etc.
Responsible for data architecture including sources, table structures, physical models
Works closely with Architects to align systems, tools and applications being utilized with business use case and performance requirements
Communicates with end users to set expectations and ensure alignment around data accuracy, completeness, timeliness and consistency
Provides data product support and maintenance
Establishes, tracks and monitors KPIs related to specific data products and deliverables

Preferred Skills and Education: Bachelors's degree in Computer Science or related field
Certification in Spark, Azure or other cloud platform
Thanks Ramesh

Related jobs

  • We work alongside the best and brightest to do incredibly cool things you wouldn\'t believe. At Roush, you\'re part of building the future. Are you someone that has a passion for providing innovated solutions to complex challenges? Do you want to work someplace where creativity and new ideas are encouraged? If so, then keep reading.

  • +-+ | Interact with customer and understand customer requirement to verify | | customer database against internal Bill of Material database for | | accuracy of nomenclature and deliver data to customer. Produce a | | cross-functional Bill of Material for benchmarking automotive | | vehicles on in-house developed Data Acquisition System (DAS Tool) and | | deliver to global teams. Taking care of the Bill of Material process, | | deliver and provide  further assistance to the customer. Coordinate | | with cross-functional team and release Bill of material for parts and | | assemblies. Ensure complete and consistent inputs and deliverables | | for schedule adherence process planning application. Ensure | | deliverables in terms of timing and quality in compliance with SLA. | | Providing  support on Data Acquisition System and create engineering | | change management solution from automotive benchmarking data. To be a | | part of Value Analysis/Value Engineering sessions to generate cost | | saving ideas based on but not limited to cost, mass and manufacturing | | processes. Report findings daily to the Value Analysis/Value | | Engineering team. Generate reports of mass analysis, material types | | and manufacturing methods to compare vehicles across multiple | | platforms. Maintain, track and create weekly update reports for Bill | | of Material, Mass Analysis and Value Analysis/Value Engineering | | reports. Assist in coupon cutting from tactical locations on the | | vehicle and perform hardness testing of material. Apply metrology | | knowledge in reverse engineering using software such as IMAGEWARE, | | GEOMAGIC STUDIO & GEOMAGIC QUALIFY. Create Metrology and Measurement | | reports adhering to Geometric Dimensioning and Tolerancing standards. | | Create 2-D sketches and 3-D models in adherence with ASME Y14.5 and | | IS 15054 standards of geometric tolerancing. Assist in taking | | technical images of automotive parts in Isometric orientation and | | Rotational Angle views. Practice continuous improvement strategies to | | eliminate waste and increase efficiency of operations. Assist in | | maintaining six sigma techniques , develop and maintain technical | | documentation & project reports. | | | | Bachelor\'s Degree in Mechanical or Manufacturing Engineering with | | five years\' experience required, foreign equivalent accepted, 40hrs a | | week. Must be willing to travel/relocate to client sites. Please send | | resume to: HR manager Caresoft Global Technologies, Inc.; 31770 | | Enterprise Drive, Livonia MI  48150. | +-+

  • Roush

Job Details

  • ID
    JC8939867
  • State
  • City
  • Job type
    Contract
  • Salary
    N/A
  • Hiring Company
    Halcyon Solutions, Inc.
  • Date
    2021-01-31
  • Deadline
    2021-04-01
  • Category

Jocancy Online Job Portal by jobSearchi.