We are looking for a Data Engineer that will work on the collecting, storing, processing, and analyzing of huge sets of data. The primary focus will be on choosing optimal solutions to use for these purposes, then maintaining, implementing, and monitoring them. You will also be responsible for integrating them with the architecture used across teams in our group.
AZURE & PYTHON EXPERIENCE REQUIRED
Responsibilities
Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities.
Implementing ETL process
Monitoring performance and advising any necessary infrastructure changes
Skillsand Qualifications
Knowledge of various persistence (RDBMS, noSQL, HDFS, Cassandra, Redis)
Understanding Data Catalog, Data Governance, Data Lineage
Experience with security, authentication in data platform
Bachelor' s Degree or above (Computer Science, Bio Engineering, Electronics and Electrical Engineering or any related field)
5+ years of experience in data engineering
Proficient understanding and experience in one or more programming languages (Python or Scala.)
Experience with building stream-processing systems, using solutions such as Spark-Streaming or Flink
Experience with Azure Databricks
Experience with integration of data from multiple data sources
Experience with building data lakes and data warehouses by leveraging any of the major cloud providers (Azure)
Familiar with Hadoop ecosystem (HDFS, HBase etc.), especially Spark
Good knowledge of Big Data querying tools
Knowledge of various ETL techniques
Knowledge of Spark MLlib
Knowledge of messaging systems, such as Kafka or RabbitMQ
To be great at our company, we find our team members have these things in common:
Gain energy from working in a fast-paced, creative environment
Decision making that employs a blend of data-driven insights and intuition
Ability to multitask and handle multiple projects concurrently
Resilience and positivity, able to address setbacks and bounce back quickly
Resourcefulness, discovering creative ways to get things done
Joy in making an immediate and positive impact
Diverse interests that are welcomed in and extend beyond our organization
Primary Location: US-TX-Dallas
Meet Your Recruiter Davis Watkins Text me about this job -Please include your name and Job Title in your Text. Thanks!