Job Description

Hybrid work mode in Madrid. This service could imply a willingness to travel occasionally within Italy.

The main responsibilities are:

Migrate the existing Hadoop infrastructure to cloud infrastructure on Kubernetes Engine, COS, Spark as a service, and Airflow as a service.
Implement data transformation and quality to ensure data consistency and accuracy. Utilize programming languages such as Scala and SQL and tools like Spark for data transformation and enrichment operations.
Set up CI/CD pipelines to automate deployments, unit testing and development management.
Write and conduct unit and validation tests to ensure accuracy and integrity of code developed.
Automate data pipelines and streamline data ingestion through the implementation of different orchestrators and scheduling processes (Airflow as a Service mainly).
Writing technical documentation (specifications, operational documents) to