Data Quality Engineer, MPID Team
SHOPEE IP SINGAPORE PRIVATE LIMITED
Job Description
Job Description:
Design, build, and maintain layered Hive/Spark SQL data models across ODS, DWD, DWS and ADS, including shared dimension tables. Define each table's grain, keys, business-date and partition logic, metric definitions, dependencies, and backfill and release procedures.
Develop and operate production batch ETL pipelines on Spark and Hive, including partner and source onboarding, incremental loads, scheduled runs, and historical backfills. Use staging and checkpoint patterns where needed so retries do not corrupt published data or SCD2 history.
Plan and execute data-store or platform migrations, including Hive-to-ClickHouse migrations where required. Design partition strategies, write migration scripts, reconcile source and target data, and validate query behavior across regions.
Build reusable cross-region models and parameterized pipelines for onboarding markets, partners, and business lines. Standardize schemas and metric logic to reduce