Overview
HCLTech Intelligent Ingestion solution helps address the demands of effectively implementing ETL processes to simplify complexities and effectively provision scalability, maintainability and governance of data workloads in AWS environment. Using AWS native services, this solution entirely automates ETL ingestion upon a single-click/event trigger and provides reusable ETL workflows triggered via AWS Step Functions parameterized to collect the required data source details that can be leveraged to bring in any supported AWS Glue data sources into raw s3 bucket or desired target data-warehouse for further data quality validation, curation and enrichment.
Key Features
Schema evolution
Fully managed, ensures data integrity during schema changes, enhances traceability between different schema versions, notifies the schema changes using the dynamic alert mechanism, schema changes do not impact existing ETL workflows
Fully automated ETL pipeline
Highly scalable and elastic to handle varying data workloads. AWS Glue and Amazon Kinesis support a wide range of data sources and data type formats
Reusable ETL workflow and transformation
Promotes code reusability, saving considerable development efforts
Centralized data catalog
Greater control of the ingested data with complete lineage for audit and compliance
Data quality (DQ) and governance
Improves data accuracy with extensive data quality monitoring with AWS Glue DataBrew
Dynamic alert mechanism, incident reporting and management
It is fully integrated with HCLTech iONA (iAct) solution to auto-create incidents in the ServiceNow tool for failure events happening at any ETL layer, including ingestion, DQ, curation, enrichment stages with appropriate email alerts to the authorized users