Job Description

Role Overview:
- Rebuild T-SQL logic as typed, tested PySpark pipelines.
- Prove that new pipelines match legacy outputs.
- Build FastAPI services that run and report on the pipeline jobs.
- Add provider-neutral LLM features into the data workflows.

Pipeline Responsibilities:
- Build Spark and PySpark pipelines on Delta Lake.
- Re-express T-SQL logic in Spark SQL.
- Ensure schema evolution, time travel, and idempotent writes using Delta Lake.
- Optimize partitioning and performance for production Spark jobs.

API and Service Development:
- Build REST APIs with FastAPI for pipeline jobs.
- Implement request validation, authentication, and job-status semantics.
- Ensure idempotency in service architecture.

Code Quality and Standards:
- Write type-hinted, tested Python code using Python 3.12, Pydantic, and static analysis tools like pyright or mypy.
- Implement automated testing with pytest, including