Company description / JD introduction

• An engineer in this role designs, develops and deploys big data projects into production

• Can handle large sets of data from different sources and make structured and meaningful data with help of the big data tools that drive business insights

   

  Responsibilities/ Tasks

• Data exploration from different streams – collecting, managing and converting raw data into information
• Understanding of data as per business requirements
• Build the infrastructure required for optimal data extraction, transformation and loading from a wide variety of data sources using SQL and Azure technologies
• Identify, design and implement internal process improvements
• Optimize data delivery and re-design infrastructure for greater scalability
• Create and maintain optimal data pipeline architecture
• Develop and implement data analysis, data collection systems and other strategies that optimize quality

• Ensure data quality and timely data availability based on SLAs

   

  Key skills required

• Hands-on experience in Pyspark or Spark and Scala
• Must have experience in cloud infrastructures like MS Azure, Azure Logic App, Azure functions, Azure storage, Security, Data Lake, Data Bricks, Synapse, etc.
• Extensive experience designing, developing and deploying big data projects into production
• Experience fixing S360 issues