Data Engineering Lead

Role Overview

1. Architect scalable data solutions using Fabric components like Lake houses, Pipelines, and Dataflows
2. Lead data ingestion and transformation using tools such as Azure Data Factory, PySpark, SQL, and Kusto Query Language (KQL)
3. Ensure data quality and governance across Bronze (raw), Silver (cleansed), and Gold (curated) layers
4. Collaborate with cross-functional teams including analytics engineers, architects, and business stakeholders
5. Optimize performance and scalability of the data platform through monitoring and tuning
6. Mentor junior engineers and guide best practices in data engineering
7. Design and implement data models and schemas to support business requirements. Ensure data accuracy, consistency, and reliability. Optimize data structures for performance and scalability.
8. Good experience on Azure Purview integrating data pipelines with Purview for Data lineage, experience with Microsoft Fabric, migrating the application from Azure Data Factory and Azure Synapse to Microsoft Fabric
9. Implement robust data security measures, including encryption, access control, and data masking. Ensure compliance with data privacy regulations and company policies.
10. Monitor and optimize data pipelines and queries for performance and efficiency. Implement caching, partitioning, and indexing strategies. Troubleshoot and resolve performance issues.
11. Establish and enforce data quality standards. Implement data governance practices, metadata management, and data lineage tracking. Ensure data quality through validation and cleansing processes.
12. Collaborate with business stakeholders to understand data requirements and deliver.
13. Create and maintain comprehensive technical documentation, including system architecture, design documents, and deployment procedures. Ensure knowledge sharing within the team.
14. Implement Lean daily management and Lean continuous improvement concepts in Application development and operations.

Required Qualifications

1. Bachelor's or master's degree in computer science, Information Technology, or a related field.
2. Proven experience in Data integration and orchestration using Microsoft Fabric tools, Cloud platforms, especially Azure (Data Factory, Synapse, Power BI) and ETL/ELT pipelines, lakehouse architecture, and real-time analytics
3. Strong foundation in SQL, PySpark, and Kusto Query Language (KQL)
4. Excellent problem-solving skills and attention to detail,
5. Strong communication and leadership skills.
6. Relevant certifications (Microsoft Certified: Fabric Data Engineer Associate) are a plus.

Preferred Qualifications

1. Effective Communicator: Possess the ability to articulate complex technical concepts in a clear and concise manner, facilitating understanding across a wide range of audiences.
2. Solution-Oriented Mindset: Demonstrated ability to approach challenges with a solutions-first attitude, proactively identifying and addressing potential obstacles.
3. Change Advocate: Capable of spearheading and implementing change, showcasing a proactive approach to driving improvements and leading teams towards new initiatives.
4. Leadership Qualities: Show a strong capacity to lead, influence, and guide teams, fostering a collaborative and productive work environment.
5. Technical Translator: Excel at translating and breaking down intricate technical topics, ensuring that stakeholders at all levels have a clear understanding of the subject matter.
6. Analytical Thinker: Possess an analytical and detail-oriented mindset, critically evaluating information derived from multiple sources and drawing meaningful conclusions.
7. Team Player: Exhibit a strong willingness to collaborate and work cohesively with colleagues, understanding the importance of collective growth and knowledge sharing.
8. Adaptability: Demonstrate resilience and adaptability in an ever-changing environment, staying updated with the latest technological advancements and best practices in the field of database administration.