Overview
Our client is among the largest telecommunication service providers in the US, offering a broad selection of smart phones, accessories and home broadband routers. They previously relied on multiple data sources, both digital and non-digital, for data analytics and reporting. Analysts manually aggregated this information to create custom dashboards according to business user needs. Our client wanted to reduce data acquisition and consolidation costs due to the diverse and complex nature of data sources.
HCLTech implemented a Databricks Data Intelligent Platform which provided a unified interface for ingesting data from different source data, provide seamless application maintenance and data management. The built-in ETL tools of the platform provided automation and scalability. The Databricks workflows enhanced speed, efficiency, accuracy and flexibility, delivering insights at a more competitive cost. HCLTech built a generic data ingestion pipeline which integrated with Adobe Experience Platform (AEP) and other data sources on Databricks Lakehouse. With Databricks, our clients could more effectively manage reporting and auditing tasks by using Databricks notebooks, where SQL and Python scripts could be seamlessly integrated with SPARK. This enabled the client to manage big data for performing analytics and building data models for informed decision-making.
The Challenge
Data migration from siloed platforms to a modern unified open data cloud architecture
Our client previously used multiple cloud data platforms for analytics, which required specific skills to handle the data workloads. Additionally, data silos existed which compounded the complexities for performing time-boxed analytics tasks. The other challenges included:
- Onboarding of new data sources from multiple source systems which took lot of time.
- Previously our client missed deadlines for time-boxed events (launching of new products in the market and modifying user tracking segments on website) resulting in higher time to market.
- There wasn’t a unified data platform with a standardized way of showcasing processed data.
- Recurring manual data validation framework was inefficient and a time-consuming activity.
- Cost increases due to manual processes. There was an urgency for including automation to integrate data silos.
- Multiple data sources for ingestion (necessary for analytics) lacked data governance standards.
The Objective
Cost optimization, automation and simplified data management using Databricks Data Intelligent Platform
In the absence of an identified baseline schema for the various data sources, our client's primary objective was to use a unified platform which offered a common interface for ingestion of different data sources. Furthermore, their sales, marketing and finance teams wanted to monitor sales and marketing campaign performance and derive timely insights during product launches to track customer’s purchasing behavior and preparing for next product launch. Lastly, they were looking to optimize their costs for hiring skilled resources to handle multiple data sources.
The Solution
HCLTech delivered a unified data platform for informed decision making
HCLTech implemented the Databricks Data Intelligent Platform to provide data migration solutions to the client. The unified platform interface enabled onboarding different data sources without any redevelopment efforts. It allowed seamless application maintenance and handled data management tasks including integration, metadata handling, automated workflows, analytics and dashboard visualization. The built-in ETL tools enabled efficient data extraction, transformation and loading, while providing automation and scalability.
HCLTech built a generic data ingestion pipeline for Adobe experience platform (AEP) and other data sources on Databricks Lakehouse which could be configured to connect any data source while dynamically performing data transformations and ingestion into the Databricks Lakehouse target system. We built data models like anomaly detection for informed decision-making. This enabled our client to analyze customer behavior on the client’s website for performing sales and marketing analytics.
The customized platform built by HCLTech could perform validation checks in Databricks to ensure user behavioral data quality is not compromised. Error detection was done from the source system automatically using push notifications. We built self-serving reporting dashboards for providing better insights to the business users while encrypting Personal Identification Information (PII). Scheduled monitoring of pipeline status dashboard also enabled tracking of the ETL process.
Our client valued the benefits of Databricks unified data source and its lake-house architecture, which combined the features of data warehouses and data lakes. With Databricks, our clients could more effectively manage reporting and auditing tasks by using Databricks notebooks, where SQL and Python scripts could be seamlessly integrated with SPARK.
The Impact
Transitioning to Databricks helped our client realize efficiency and performance gains within 90 days
HCLTech recommended Databricks Data Intelligent Platform for its inbuilt capability to perform accurate analytics.
- Databricks contributed to a 70%-80% gain in time and cost efficiencies
- 2x-3x increase in accuracy, decision making and performance with Delta Lake implementation on Databricks
- 2x-3x faster reporting could be done using Databricks by automating workflows and building generic pipelines
- By leveraging the cloud-based architecture, they were able to process millions of records in just a few hours using Databricks' data lake architecture, compared to the days or weeks previously spent on manual reports with other tools
- Transitioning to Databricks Delta Lake resulted in 2x-3x cost reduction by consolidating organizational data
