Job Description :
- Analyses current business practices, processes, and procedures as well as identifying future business opportunities for leveraging Microsoft Azure Data & Analytics Services.
- Provide technical leadership and thought leadership as a senior member of the Analytics Practice in areas such as data access & ingestion, data processing, data integration, data modeling, database design & implementation, data visualization, and advanced analytics.
- Engage and collaborate with customers to understand business requirements/use cases and translate them into detailed technical specifications.
- Develop best practices including reusable code, libraries, patterns, and consumable frameworks for cloud-based data warehousing and ETL.
- Maintain best practice standards for the development or cloud-based data warehouse solutioning including naming standards.
- Designing and implementing highly performant data pipelines from multiple sources using Apache Spark and/or Azure Databricks
- Integrating the end-to-end data pipeline to take data from source systems to target data repositories ensuring the quality and consistency of data is always maintained
- Working with other members of the project team to support delivery of additional project components (API interfaces)
- Evaluating the performance and applicability of multiple tools against customer requirements
- Working within an Agile delivery / DevOps methodology to deliver proof of concept and production implementation in iterative sprints.
- Integrate Databricks with other technologies (Ingestion tools, Visualization tools).
- Proven experience working as a data engineer
- Highly proficient in using the spark framework (python and/or Scala)
- Extensive knowledge of Data Warehousing concepts, strategies, methodologies.
- Direct experience of building data pipelines using Azure Data Factory and Apache Spark (preferably in Databricks).
- Hands on experience designing and delivering solutions using Azure including Azure Storage, Azure SQL Data Warehouse, Azure Data Lake, Azure Cosmos DB, Azure Stream Analytics
- Experience in designing and hands-on development in cloud-based analytics solutions.
- Expert level understanding on Azure Data Factory, Azure Synapse, Azure SQL, Azure Data Lake, and Azure App Service is required.
- Designing and building of data pipelines using API ingestion and Streaming ingestion methods.
- Knowledge of Dev-Ops processes (including CI/CD) and Infrastructure as code is essential.
- Thorough understanding of Azure Cloud Infrastructure offerings.
- Strong experience in common data warehouse modeling principles including Kimball.
- Working knowledge of Python is desirable
- Experience developing security models.
- Databricks & Azure Big Data Architecture Certification would be plus
Mandatory skill sets:
ADE, ADB, ADF
Years of experience required:
3-7 Years
Education qualification:
BE, B.Tech, MCA, M.Tech