Lead Data Engineer
EPAM Systems
AI assistant
Job Description
We're seeking a hands-on Lead Data Engineer to construct and maintain scalable data pipelines along with cloud-based data solutions. The right person for this role brings substantial background in Python, PySpark, AWS, Airflow, and DevOps, with particular attention to production-grade engineering and automation practicesResponsibilitiesCreate, construct, and sustain scalable data pipelines running on AWS infrastructureBuild dependable data processing solutions leveraging Python, Pandas, and PySparkEstablish and oversee workflows via AWS Managed Workflows for Apache Airflow (MWAA)Craft and fine-tune data queries through AWS Athena to enable efficient analytical workStreamline infrastructure setup and deployment workflows through automation using Terraform and GitHub ActionsPut in place testing, monitoring, troubleshooting practices, and data-quality safeguards to keep pipelines dependableTake part in reviewing code and help promote sound engineering practices throughout the teamWork