Create and maintain optimal data pipeline architecture for enterprise big data and analytics in a cloud based big data environment
Design and assemble large, complex data sets that meet functional / non-functional business requirements.
Implement internal process improvements in data analytics: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
Build the infrastructure and workflows required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and ‘big data’ technologies.
Build analytics applications that utilize the data pipeline to provide actionable insights into key business performance metrics.
Create tools and processes to keep sensitive data, including PHI and PII data separated and secure across clients.
Work with data and analytics experts to strive for greater functionality in our data systems to help make us a market leader in big data.
Qualifications:
5+ years of Experience with Azure Data Factory, SSIS, SAS, Informatica or other ETL tools including building and optimizing big data pipelines and data sets
5+ years of experience with Power BI or other similar tools
3+ years of Experience on Azure Databricks with Python and PySpark
3+ years of Experience on frameworks such as .NET and .NET Core
1+ years of Experience in the Azure Data Engineering environment using Databricks, Azure Data Lake Storage, etc.
Expert in using XML, JSON, EDI, or other data schema formats
Expert in object-oriented analysis, SOA, design patterns, development, implementation and maintenance
Expert with relational databases (SQL Server preferred) and query languages such as SQL, T-SQL, Linq, etc.
Very good knowledge of Microsoft development tools such as
Languages like C#, Visual Basic .NET, F#, C .NET
Web frameworks like ASP.NET, MVC 1-6, ASP.NET Web Api
Messaging and data exchange using frameworks like WCF, ASP.NET WebApi
ORM’s such as Entity Framework
Experience in agile devops including continuous integration
Unit testing frameworks nUnit, SpecFlow, MSTest, xUnit
Source code control using Git and GitFlow
Code deployment using tools like TeamCity, Jira, etc.
Agile team workflow using SAFe
Excellent verbal and written communication skills
Experience in onshore/offshore mixed development team models
