Job Responsibility:
Architect and design solutions to meet functional and non-functional requirements.
Lead the design, implementation, and optimization of our Databricks platform.
Work closely with our data engineering team to ensure that our Databricks platform is optimized for performance, scalability, and reliability.
Develop and maintain a comprehensive understanding of our data pipeline and data architecture.
Additional Skills:
- 9-15 years experienced resources in Databricks, Python, Big Data, Apache Spark, SQL and Spark SQL
- Strong hands on in Pyspark and Apache Spark.
- Experience in Building Data Governance Solutions like Unity Catalog …etc
- Build Very Strong Orchestration Layer in Databricks/ADF…. Workflows.
- Build CICD for Databricks in Azure Devops.
- Process near Real time Data thru Auto Loader, DLT Pipelines.
- Implement Security Layer in Delta Lake.
- Implement Massive Parallel Processing Layers in Spark SQL and PySpark.
- Implement Cost effective Infrastructure in Databricks.
- Ability to Build Solutions as Cloud Agnostic.
Skills:
- Databricks
- Spark
- Azure