Job description:

To be responsible for managing technology in projects and providing technical guidance solutions for work Job Summary:

We are seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. The ideal candidate will have a strong background in developing, and delivering scalable data solutions using Azure Databricks, Azure Data Lake, and Azure Cosmos DB. This role will involve coaching data engineers technically on topics like Azure databricks, optimizing data workflows, and collaborating with internal teams on various aspects.

Responsibilities:

1. Develop, and implement scalable data pipelines using Azure Databricks and Azure Data Lake.
2. Create and maintain ETL workflows to ensure data quality, integrity, and availability.
3. Optimize Azure Cosmos DB schemas and queries for high performance and cost-efficiency.
4. Demonstrated ability to manage and analyze data at scale within the Azure ecosystem.
5. Coach junior data engineers on technologies like Azure databricks, Azure Cosmos database so the candidate should be proficient in such topics with greater amount of hands-on experience.
6. Implement CICD pipelines using Azure DevOps to streamline deployment processes.
7. Conduct code reviews, provide mentorship, and facilitate knowledge sharing sessions within the team.
8. Integrate data from various sources into a centralized data lake, enabling unified access and analysis.
9. Perform data cleansing, transformation, and migration tasks as required.
10. Monitor and troubleshoot data pipelines and resolve any issues promptly.

Requirements:

1. Bachelor s or Master s degree in Computer Science, Information Technology, or a related field.
2. 7+ years of experience in data engineering or related roles.
3. 5+ years of experience in Azure Databricks, Azure Data Lake, and Azure Cosmos DB proven track record of building data engineering solutions
4. Proficient in programming languages such as Python, SQL, and Scala.
5. Experience with big data technologies including Apache Spark, Hadoop, and Hive.
6. Hands-on experience with cloud platforms, particularly Microsoft Azure.
7. Proficiency in ETL processes, data warehousing, and data modeling.
8. Proficient in using Azure Cosmos DB for graph-based data analysis.
9. Knowledge of tools and technologies such as Azure Data Factory, Azure Synapse Analytics, and Apache Kafka.
10. Solid understanding of CICD pipelines and hands-on experience with DevOps tools such as Git for continuous integration and deployment.
11. Excellent problem-solving skills and the ability to troubleshoot complex data issues.
12. Strong coaching abilities, Excellent communication and collaboration skills, with the ability to work effectively with cross-functional teams.

Preferred Qualifications:

Microsoft Certified: Azure Data Engineer Associate.
Databricks Certified Associate Developer for Apache Spark.
Experience with data governance and data security practices.
Knowledge of Docker and Kubernetes for containerization and orchestration.

Please find the below points to cover in addition to the job spec for senior data engineer contract role

1. Design develop comprehensive data models (logical physical) to meet business requirements, ensuring data structures are optimized for performance & efficiency by working closely with the data lead
2. Implement data governance & ensure data integrity by applying appropriate modelling techniques, including the use of star & snowflake schema for data warehousing.
3. Work with cross functional teams to integrate data models into large scale data engineering solutions, leveraging cloud platforms like Azure

• To be responsible for providing technical guidance / solutions ;define, advocate, and implement best practices and coding standards for the