HCLTech’s marquee client is an American financial services company specializing in banking and insurance services. We are deeply involved in all aspects of the client’s operations and facilitating their transition from Netezza and Sailfish to Snowflake through cloud-based ELT (Extract Load Transform) solutions. Learn how HCLTech started a data modernization journey for our client, developed technical assets like solution frameworks and reference architectures to migrate the client’s current analytical on-premises platform to a cloud-based Snowflake environment. The aim was to enhance member experience, reduce costs and improve analytical performance by leveraging open-source and Software-as-a-Service (SaaS) offerings on cloud.
The Challenges
Key challenges driving migration to snowflake environment
Our client’s IT team opted to migrate their current on-premises analytical platform to a cloud-based Snowflake environment to tackle several challenges:
- Delays in data refresh were hindering business reporting and decision making processes
- The legacy Netezza database had reached its maximum storage threshold, leading to degraded query performance
- The existing infrastructure couldn’t support storage extension, prompting manual interventions by data administrators
- Concurrency issues arose when numerous queries competed for resource availability
The Objective
Achieving data migration and performance optimization with Snowflake
Our client prioritized technology transformation by migrating data from Netezza to Snowflake using dbt cloud ELT tool. This initiative standardized data migration and improved timely data consumption and performance for business reporting, enhanced data lineage, data quality and data security to meet compliance and regulations. We assisted in resolving scalability and performance issues by leveraging Snowflake to optimize costs.
The Solution
Technology transformation: Staged data layer approach for transitioning from on-prem to cloud
The migration from on-premises to cloud was strategically planned with a staged data layer approach.
- Staging layer (Stage 0) - Data in this layer contained sensitive information and was not suitable for direct business consumption. It consisted of data from various sources stored in raw form.
- Raw-Conformed layer (Stage 1) - This data which was tokenized, was partitioned according to standards and stored in AWS S3. This wasn’t suitable for direct business interactions.
The Snowflake Cloud Data Platform was customized to handle enterprise data and analytics across a hybrid infrastructure.
- In Stage 2 (Raw-Query layer) - Data Scientists conducted intensive computation and Machine learning analytical queries
- In Stage 3 (Integrated Data Layer) - The integrated data layer facilitated data integrations between on-premises and cloud data, enabling queries was done for queries on any data transformation or analytical tasks
- In Stage 4 (Question Focused Layer) – Data in this layer was readily accessible for business consumption and reporting according to specific use cases
HCLTech used Snowflake for initial raw data handling, enabling data scientists to perform computation and ML queries. Snowflake was also used to integrate on-premise and cloud data to facilitate business consumption and reporting.
The Impact
Unlocking benefits with a technology transformation using Snowflake and cloud solutions
- Boosted data refresh and report query performance by 30%
- Cloud migration yielded 25% long-term cost savings for on-premise applications
- Effortless compute and seamless storage resource management was possible with Snowflake on AWS
- Role-based security was enabled for data management across multiple domains
- Snowpark was leveraged in the feature engineering and AI/ML area
- In future ML endeavors, we can utilize container service and cortex
- We harnessed boundless on-demand scalability and computational capabilities and prowess via Snowflake and Cloud platforms
- Eliminated the need for maintaining On-Premises applications
