Overview

In today's digital age, the volume of data generated daily is staggering, reaching approximately 5 quintillion bytes worldwide. Businesses strive to leverage this wealth of information to stay ahead in the market. However, a critical challenge arises amidst this data deluge: ensuring stakeholders have access to high-quality data for informed decision-making.

Deciphering the data landscape

Consider a leading financial institution in North America grappling with myriad issues within their data and analytics platform on Google Cloud Platform (GCP). These pain points are emblematic of broader challenges faced by organizations navigating the complex landscape of data management:

1. Fragmented data: Data is scattered across various locations and departments, hindering seamless cross-referencing and decision-making
2. Dataset discovery dilemma: The absence of a cohesive schema or data catalog makes finding relevant datasets difficult, complicating data utilization
3. Governance conundrum: Enforcing governance standards across diverse datasets requires robust access control and monitoring protocols
4. Data lineage labyrinth: The lack of comprehensive data lineage mechanisms impedes root cause analysis

While the idea of a tailored solution within the Google Cloud environment may be appealing, a more practical approach is necessary due to increasing complexities. It's crucial to explore streamlined methods for overcoming these challenges, enabling improved data accessibility, governance and lineage tracking without relying on bespoke solutions.

Our way forward

The principal objective of the financial institution is the unification of data across various sources and creating a single pane of glass that brings their data together with security and governance. We were sure Goggle Dataplex was the right choice to address the challenges. Google Dataplex is an intelligent data fabric offering that enables organizations to centrally discover, manage, monitor and govern data across multiple sources.

The following are key components of Dataplex.

Implementation approach

Dataplex configuration and setup:

Mapping of assets and enabling the data discovery

By onboarding assets under their respective zones, which can be GCS (Google Cloud Storage) or BigQuery datasets, Dataplex automatically generates metadata and enables data discovery. This allows seamless integration of GCS buckets and BigQuery datasets using SQL in Dataplex Exploration Workbench. As a result, we achieved structured data organization for various services, seamless data exploration across lakes and zones and reduced time to market for operational efficiency. Below is the structure representing the layers created.

Fine-grained access control

We established principles and allocated resource access based on roles, leveraging Dataplex's flexibility to configure access at both the resource and principal levels. Utilizing IAM (Identity and Access Management), we managed principles and implemented fine-grained access control at both the DATA and LAKE levels. This approach enabled granular-level access control and seamless integration and IAM policies, facilitating role-based access control and fulfilling the financial institution's requirement to restrict cross-zone data exploration.

Data processing

We leveraged Dataplex's built-in templates for various data engineering tasks such as data preparation, ingestion, data quality checks and dataflow pipelines. Additionally, Dataplex supports running custom Dataproc serverless tasks, all orchestrated centrally through Composer. By incorporating these tasks into our end-to-end data processing, we achieved improved performance, seamless orchestration and monitoring through Composer. Furthermore, we seamlessly integrated data quality (DQ) processes with existing Composer pipelines, enhancing the overall efficiency of our data workflows.

Data quality

Dataplex provides Auto data quality (AutoDQ) and Dataplex data quality tasks to validate data quality:

1. Auto data quality (AutoDQ): AutoDQ automates and simplifies quality definition with recommendations and UI-driven workflows. We can create rules for different dimensions and configure the rule execution for incremental data and full data with a threshold limit.
2. Dataplex data quality task: The Dataplex data quality task uses an open-source component, CloudDQ, which can also open up choices for customers who want to enhance the code to their needs. The rules can be configured in a YAML file and referenced from a GCS bucket.

Our design approach encompassed several key elements:

1. Custom DQ tasks integrated with Composer for flexibility in workflow orchestration
2. AutoDQ was leveraged for selected sources, optimizing efficiency
3. Endpoints were established for programmatic data consumption, with data persistence in BigQuery storage and Cloud Logging sinks, and support for integrations with reporting tools like Looker and Google Data Studio
4. Dataplex orchestration of execution management for streamlined operations

This approach delivered significant benefits:

1. Simplified implementation of data quality dimensions, adaptable to diverse requirements
2. Enhanced threshold percentages from 40% to 80%, improving data accuracy and reliability
3. Performance improvements and enhanced data stability facilitated faster decision-making processes, driving operational efficiency

Data profiling

We meticulously analyzed source data profiles by configuring and scheduling checks using the Data Profile module within Dataplex. This module identifies key statistical characteristics such as null%, unique% and distribution, providing relevant metrics tailored to column types (e.g., integer or string).

Our design approach involved:

1. Creating a data profiling scan linked to a BigQuery table, with the sampling size determined by the criticality of the source data
2. Utilizing Looker Studio to generate insights derived from data profiling over a specified period, enhancing data understanding and decision-making processes

In addition to data profiling, we utilized other Dataplex modules to facilitate efficient catalog management and establish governance policies, ensuring comprehensive data management practices.

Manage catalog

1. Tag templates: Utilizing tags for data entries provides meaningful context to stakeholders. For example, tags assist in identifying tables having the PII (Personally Identifiable Information) data, data quality scores, data retention policy, etc.
2. Entry groups: A traditional approach to data catalogue management, allowing users to create their own entry groups to contain their cloud storage fileset entries and the IAM policies associated with those entries that specify the users who can create, edit and view entries within an entry group.
3. Glossary: Primarily used in calculated fields, glossary is a single place to maintain and manage business-related terminology and definitions across the organization.
4. Attribute store: Used for creating taxonomies for data attributes that can be attached to datasets.

Conclusion

With its intelligent data fabric, Dataplex empowers enterprises to use their data effectively and power analytics and data science workloads. We view Dataplex as a pivotal tool in unifying data assets, enhancing data management and providing access to high-quality data. Undoubtedly, Dataplex stands poised to revolutionize data lakes and data management, creating a significant impact on enterprise-scale data solutions.

For more information on Dataplex and getting started, reach out to Digital Business team, HCLTech.