Job Description

Design and maintain scalable ETL/ELT pipelines in Azure Databricks to process structured, semi-structured, and unstructured insurance data from diverse sources.

Collaborate with architects, modellers, analysts, and stakeholders to deliver tailored data assets for analytics, operations, and regulatory reporting.

Develop and optimize batch and streaming data solutions using Azure Data Factory, Data Lake Storage Gen2, Azure Event Hubs, and Kafka.

Implement robust data validation, cleansing, and quality controls to ensure reliability for critical insurance use cases.

Integrate Informatica tools to support governance, metadata management, lineage tracking, and cataloguing.

Enforce data security best practices including RBAC, managed identities, encryption, and compliance with PDPO, GDPR, and other regulations.

Automate deployment workflows using GitHub Actions for efficient, repeatable, and auditable data operations.

Troubleshoot pipeline