Our client is looking Big Data Engineer for Long term project in Multi Location (USA) below is the detailed requirements.Job Title : Big Data EngineerLocation : Multi Location (USA)Duration : Contract / Full-TimePrimary Skill : AWS CloudPython programming and AWS services (EC2, RDS, S3, Lambda, .Airflow DAGPython to support data operations.Integrate various internal and external data sources (including SharePoint, One Drive, One Note,Understanding of CI / CD tools and practices.Job Description :Using AWS cloud services.Develop and maintain automated data pipelines, ensuring data accuracy and efficient processing using Airflow DAG.Write high-quality, maintainable code in Python to support data operations.Integrate various internal and external data sources (including SharePoint, One Drive, One Note, to create a unified data platform.Build tools for document ingestion, dynamic file previewing, and tabular data extraction.Manage graph database systems and support large text corpus management.Implement security controls to maintain data integrity and compliance.Use Large Language Models for keyword extraction and data analysis to gain insights.RequirementsProven experience with Python programming and AWS services (EC2, RDS, S3, Lambda, .Proficient in managing and orchestrating workflows with Airflow.Solid understanding of CI / CD tools and practices.Demonstrated experience with data engineering and familiarity with graph database management.Expertise in data processing, corpus management, and keyword extraction.Strong problem-solving abilities, attention to detail, and organizational skills.Excellent communication and teamwork skills to collaborate with cross-functional teams.
