Senior Data Engineer

FROSTBYTE

📍 Johannesburg, South Africa On-Site ⏳ 7h ago
AI assistant

Job Description

We are hiring a Senior Data Engineer with strong expertise in Azure cloud, Databricks, PySpark, and orchestration of data pipelines. The ideal candidate must hold a minimum of an Honours degree in an IT-related field and be experienced in building scalable, automated data solutions on Azure.

Key Responsibilities

  • Design, develop, and maintain scalable and automated data pipelines using Azure Data Factory, Azure Databricks, and Azure Data Lake.
  • Build and optimize PySpark-based data workflows in Azure Databricks for batch and streaming workloads.
  • Develop ETL/ELT processes and ensure reliable orchestration and scheduling of data workflows.
  • Implement CI/CD practices and DevOps methodologies for data pipeline deployments.
  • Ensure high availability, performance, and security of data pipelines in production environments.
  • Monitor, troubleshoot, and resolve issues in data workflows and pipeline performance.
  • Document