Data Engineer with Pyspark

Data Engineer with Pyspark

20 Jan 2024
Texas, Fortworth, 76101 Fortworth USA

Data Engineer with Pyspark

Vacancy expired!

  • The big data and predictive analytics team is seeking an individual to standardize and automate open source processes and functions.
  • Primarily, this position will focus on automating quality assurance, testing, and provisioning processes.
  • Extended areas of concentration will be creating end user applications intended to expedite and automate data acquisition, integration, and control processes.

Duties/Responsibilities:
  • Responsible for identifying and designing automation opportunities
  • Build a working knowledge of all existing operational analytics and data pipeline methods and functionality
  • Create effective and automated processes to drive efficiency and consistency across all areas of big data and advanced analytics projects
  • Work closely with peers and leadership within Technology Services to develop an automation framework
  • Create an infrastructure to support continued growth in the open source environment

Basic Qualifications:
  • 8-10 Years of experience in SQL queries and working with NoSQL, RDBMS
  • Bachelor’s in Business, Information Technology, Computer Science, Economics or other related discipline or equivalent work-related experience
  • Apache Hadoop software library knowledge
  • Proficient with automation tools and framework design
  • Experience in automation scripting and web application development
  • Strong personal and organizational skills to execute project deliverables
  • Excellent verbal and written communication skills to lead a cross functional team

Preferred Qualifications:
  • MBA or equivalent degree
  • Working knowledge of Hortonworks 2.3 or greater
  • Experience in using Java, pySpark, Python, Scala, Spark, Shell script, Spark with Python, Spark streaming and SQL.
  • Background in ETL and data warehousing
  • Sqoop, Hive and Big data concepts.
  • Shell scripting
  • ADF, Azure databricks , ADLS is good to have

Related jobs

Job Details

  • ID
    JC8378067
  • State
  • City
  • Job type
    Permanent
  • Salary
    Depends on Experience
  • Hiring Company
    Tech Mahindra (Americas) Inc.
  • Date
    2021-01-11
  • Deadline
    2021-03-12
  • Category

Jocancy Online Job Portal by jobSearchi.