Position: Lead Data Engineer -Data Bricks Location : Watsonville, CA Type: Contract Duration : 6-12 months Only 10 +Years| LinkedIn is mandate
Job Description :
Minimum 7+ years of experience in design, development, and deployment of large-scale, distributed, and cloud-deployed software services.
Bachelor’s in Computer Science or related disciplines
Must have AWS Data Engineer with Databricks and Deltalake experience.
Must have deep understanding of Spark, AWS (specifically Redshift, S3, Glue, Athena) and should be having practical exposure on executing multiple projects in Deltalake
Must have been part of minimum 2 end to end big data projects and must have handled defined modules independently.
Expert in SQL and good with data modelling for relational, analytical and big data workloads.
Advanced programming skills with Python, Scala or Java.
Strong knowledge of data structures, algorithms, & distributed systems.
Strong experience and deep understanding of Spark internals.
Expert in Hive.
Hand on experience with one of the cloud technologies (AWS, Azure, Google Cloud Platform).
Hands on experience with at least one NoSQL database (HBase, Cassandra, MongoDB etc).
Experience in working with both batch and streaming datasets.
Knowledge of at least one ETL tool like Informatica, Apache NiFi, Airflow, DataStage etc.
Experience in working with Kafka or related messaging queue technology.
Hands on experience in writing shell scripts for automating processes.
Willingness to learn and adapt.
Delivery focused and willingness to work in a fast-paced work environment.
Takes initiative and responsibility for delivering complex software.
Knowledge of building REST API end points for data consumption.
Excellent oral and written communication is a must.
Well versed with Agile methodologies and experience in working with scrum teams.
Preferred
Master's in Computer Science or related disciplines
Experience building self-service tools for analytics would be plus.
Knowledge of ELK stack would be a plus.
Knowledge of implementing CI/CD on the pipelines is a plus.
Knowledge of Containerization (Docker/Kubernetes) will be plus.
Knowledge of building RESTful services would be an added advantage.
Experience working with one of the popular Public Cloud based platforms
