Systems/Data Architect

Systems/Data Architect

Vacancy expired!

Please, no third parties.
Permanent residents only.

Main Duties & Responsibilities:
- Play an integral role with the design and development of a petabyte-scale data lake, automated data ingestion pipeline, distributed data processing systems, real-time data streams, low-latency high-volume spatial, graph and raster APIs on top of AWS infrastructure.
- Design high-performance data ingestion, storage, compute and delivery systems that serve a variety of consumers from web apps and APIs to data scientists and business intelligence.
- Play an integral role with the strategy and architecture of company's data platform, as well as for participating in all aspects of the development lifecycle.
- Interact with both technical and non-technical audiences, mentor team members in new technologies and paradigms, and contribute to the continuous improvement of systems and processes.
- Evaluate emerging technologies and provide recommendations for product improvements.
- Identity and implement new capabilities within the platform that create new opportunities for both real-time web speed queries and for long-running asynchronous analysis.
- Design elegant and intuitive REST and GraphQL APIs that support a range of use cases from basic queries to complex user-defined compute pipelines.
- Implement new models for linking and modeling datasets through spatial and relational models, machine learning, natural language processing, computer vision and other techniques, and increase the velocity of data ingestion and processing while reducing human touch points.
- Align the data platform with the use cases of its consumers, including web applications, API integrations, data scientists, BI analysts and others.
- Ensure that systems are manageable, maintainable, reliable, scalable and secure, designing best practices around infrastructure automation, cloud scaling, quality assurance, monitoring, logging, data governance, security and privacy, etc.
- Collaborate with managers of data platform engineering teams to ensure that systems are built as designed and interoperate effectively.
- Participate in team activities such as design sessions, core reviews and sprint ceremonies.
- Adhere to best practices around versioning, automated testing, dependency management, system reliability, containerization, infrastructure-as-code, auto-scaling, data security, etc.
- Contribute to company's technology strategy as a member of its architectural leadership team.

Skills & Requirements:
- 8+ years of experience in software engineering and/or systems architecture.
- 5+ years of experience building distributed big data systems, including 2+ years working in an architect or leadership capacity.
- BS/BA in Computer Science or related field is highly preferred.
- Hands-on architecture experience including designing and building distributed, cloud-native data storage (AWS), processing and delivery systems, as well as an interest in helping lead a Big Data transformation.
- Distributed data processing systems, including Spark and Dask.
- Data lake storage formats, ideally including Parquet and Hudi.
- Relational, graph and document databases systems (geospatial data, graph data, and raster data preferred).
- Search and cache layers, including Elasticsearch, Redis and Memcached.
- Low-latency models for delivering data lake data at web speed.
- Real-time data streaming systems, including Kafka.
- Data lake strategies for metadata, ontology, governance, authorization, etc.
- Microservice-based architecture and infrastructure, including with Kubernetes.
- Data ingestion automation pipelines, such as Airflow or Prefect.
- Infrastructure-as-code systems (IAC), including Chef or Terraform.
- AWS experience in solutions architecture and cost management.
- Modern practices around agile development, release management, continuous integration, system reliability, cloud architecture and data security.
- Familiarity with spatial techniques/standards such as Geohash, Quadkey, H3, WMS and WMTS.
- Familiarity with data modeling, graph theory, knowledge graphs and ontology design.
- Familiarity with AuthN/Z standards and practices, including with OAuth 2.0, SAML and OIDC.
- API design fundamentals, including REST, GraphQL and gRPC.
- Data system fundamentals such as partitioning, optimization, indexing, query planning, etc.


tags: system architect, systems architect, data architect, database architect, big data architect, cloud architect, aws architect, azure architect