All about you:
Oversee the administration, configuration, and maintenance of Databricks clusters and workspaces.
Continuously monitor Databricks clusters for high workloads or excessive usage costs, and promptly alert relevant stakeholders to address issues impacting overall cluster health.
Implement and manage security protocols, including access controls and data encryption, to safeguard sensitive information in adherence with Mastercard standards.
Facilitate the integration of various data sources into Databricks, ensuring seamless data flow and consistency.
Identify and resolve issues related to Databricks infrastructure, providing timely support to users and stakeholders.
Work closely with data engineers, data scientists, and other stakeholders to support their data processing and analytics needs.
Maintain comprehensive documentation of Databricks configurations, processes, and best practices and lead participation in security and architecture reviews of the infrastructure
Bring MLOps expertise to the table, namely within the scope of, but not limited to:
Model monitoring
Feature catalog/store
Model lineage maintenance
CI/CD pipelines to gatekeep model lifecycle from development to production
Own and maintain MLOps solutions either by leveraging open-sourced solutions or with a 3rd party vendor
Build LLMOps pipelines using open-source solutions. Recommend alternatives and onboard products to the solution
Maintain services once they are live by measuring and monitoring availability, latency and overall system health.
What experience you need:
Master’s degree in computer science, software engineering, or a similar field.
Strong experience with Databricks and its management of roles and resources
Experience in cloud technologies and operations
Experience supporting API’s and Cloud technologies
Experience with MLOps solutions like MLFlow
Experience with performing data analysis, data observability, data ingestion and data integration.
5+ DevOps, SRE, or general systems engineering experience.
2+ years of hands-on experience in industry standard CI/CD tools like Git/BitBucket, Jenkins, Maven, Artifactory, and Chef.
Experience architecting and implementing data governance processes and tooling (such as data catalogs, lineage tools, role-based access control, PII handling)
Strong coding ability in Python or other languages like Java, and C++, plus a solid grasp of SQL fundamentals
Systematic problem-solving approach, coupled with strong communication skills and a sense of ownership and drive.
What could set you apart
• SQL tuning experience.
• Strong automation experience
• Strong Data Observability experience.
• Operations experience in supporting highly scalable systems.
• Ability to operate in a 24x7 environment encompassing global time zones
• Self-Motivating and creatively solves software problems and effectively keep the lights on for modeling systems.