Apply to Data Engineer (PostgreSQL to Snowflake Migration)

Sphere is a technology and consulting company specializing in software engineering, technical recruiting, and staff augmentation. We help businesses scale their engineering teams and deliver technology initiatives through experienced technical professionals.

About the Role

We are looking for a Data Engineer to support a data migration initiative for our client. The project involves migrating approximately 200 existing ETL jobs from PostgreSQL to Snowflake.

The core platform is already established and operational. The primary focus will be adapting existing SQL logic and pipeline orchestration, validating results against the current PostgreSQL outputs, and troubleshooting discrepancies to ensure data consistency.

You will work independently on migration tasks while collaborating with the existing engineering team for onboarding and technical support.

Engagement: 3-month initial contract, full-time, with potential extension.

Responsibilities:

• Migrate and adapt existing SQL scripts and ETL jobs from PostgreSQL to Snowflake.

• Configure and update Python-based pipeline orchestration to support the migrated jobs.

• Develop validation scripts to compare outputs between PostgreSQL and Snowflake.

• Investigate and troubleshoot data discrepancies, identifying root causes related to source data, replication delays, SQL behavior, or differences between database platforms.

• Debug and resolve migration issues while preserving existing transformation logic and expected outputs.

• Collaborate with the existing engineering team to understand current pipelines, resolve technical challenges, and ensure a reliable migration process.

Requirements

• Strong SQL skills, including writing, analyzing, and debugging complex queries.

• Hands-on experience with PostgreSQL and Snowflake.

• Practical experience developing, maintaining, or migrating ETL/ELT pipelines.

• Proficiency in Python for pipeline orchestration, scripting, and data validation.

• Experience comparing datasets across systems, investigating discrepancies, and performing data reconciliation.

• Strong troubleshooting and analytical skills, with the ability to identify root causes of data inconsistencies.

• Ability to work independently, manage assigned tasks, and communicate effectively with an existing engineering team.

Nice to Have

• Experience with Pentaho or other ETL/orchestration tools.

• Experience working with custom Python-based ETL frameworks.

• Familiarity with data replication, CDC, and replication lag.

• Previous experience migrating SQL workloads between relational databases and cloud data warehouses.

Project Details

• Duration: 3 months initially, with potential extension.

• Workload: Full-time.

• Collaboration: Work alongside the client's existing engineering team.