segment-pixel
For the best experience, try the new Microsoft Edge browser recommended by Microsoft (version 87 or above) or switch to another browser � Google Chrome / Firefox / Safari
OK
brand-elements brand-elements brand-elements brand-elements brand-elements
brand-elements

Client Background

Our client provides cloud-based survey development and marketing analytics solutions for a wide range of industries, including 98% of the Fortune 500. As part of a data infrastructure modernization initiative, the company wanted to migrate the legacy SQL server data warehouse to SaaS-based Snowflake. Xoriant’s expertise in database migration and big data testing was leveraged to migrate the data and ETL scripts with optimal speed and accuracy.

The key objectives included:

  • Migrate 500+ ETL scripts, 1300+ DB objects, ~9TB data to Snowflake.
  • Ensure testing accuracy up to two decimal points for all migrated data and ETL scripts.
  • Expedite data migration and testing process to meet stringent timelines.
  • Create a parallel Snowflake environment to compare/validate SQL and Snowflake data.

Xoriant Solution | Key Contributions

The Xoriant team assessed the migration needs and scope with respect to timelines. The analysis revealed that although the SQL to Snowflake code migration utilized multiple accelerators, data migration and continuous data testing between the SQL server and Snowflake would require test automation to achieve the desired speed and accuracy. To accelerate migration and testing, the Xoriant team came up with a flexible, metadata-driven solution framework which acted as a data validation accelerator.

Solution features included:

  • Test Automation Framework
    • The Test Automation framework was designed to ensure the table metadata and test criteria were captured in the framework tables.
    • Metadata enabled manual and scheduled execution of test cases.
    • For ongoing regressions, the readily available metadata enabled faster execution of scheduled automated testing.
  • Objects and Data Migration
    • Migrating table structures from SQL to Snowflake (with/without data).
    • Scheduling to enable migration of data during off-hours.
  • Quality Assurance – Table Structure and Data Validation
    • Metadata-driven parameters facilitated testing based on data types (counts, distinct, aggregate).
    • Compared the table structure and data using the automation solution with two decimal-point accuracy.
    • Scheduled and automated data validation during off-hours.
    • Executed continuous regression testing on schedule.
    • Published a data validation report with sample records of unmatched datasets for analysis.
  • Parallel Environment for Data Comparison
    • Ran data flows in parallel for a specific period to compare the quality and accuracy of migrated history data and daily delta validation.
    • After migration, automation tools are still being used for faster change requests and enhancement data and structure validation. 

High Level Architecture

Big Data Testing Diagram - Xoriant Case Study

Key Benefits

  • Delivered 50% faster migration and test coverage with metadata-based testing.
  • Accelerated validations by testing 1300+ objects in a parallel environment.
  • Ensured seamless conversion and testing of 500+ SQL scripts to Snowflake queries.
  • Migrated 1000+ table structures and history data to Snowflake.
  • Improved efficiency by 60% using test automation, resulting in meeting strict migration timelines.
  • The custom test automation framework enabled the client to achieve its strategic migration goals.

Technology Stack

SQL Server | Snowflake | Python 3.8 | Airflow

Get Started

arrow arrow
vector_white_1
Think Tomorrow
With Xoriant
triangle triangle triangle triangle triangle
Is your digital roadmap adaptive to Generative AI, Hyper cloud, and Intelligent Automation?
Are your people optimally leveraging AI, cloud apps, and analytics to drive enterprise future states?
Which legacy challenge worries you most when accelerating digital and adopting new products?

Your Information

12 + 3 =
Solve this simple math problem and enter the result. E.g. for 1+3, enter 4.

Your Information

15 + 2 =
Solve this simple math problem and enter the result. E.g. for 1+3, enter 4.

Your Information

3 + 2 =
Solve this simple math problem and enter the result. E.g. for 1+3, enter 4.