Role Requirements:
- No ML/AI development skills are needed, the data science team will do modifications to models. Experience supporting operating environments for ML/AI will be a bonus.
- Need cloud Data Engineering experience – Need to be experienced enough to operate independently to identify, debug, and repair issues; still expect to work together to triage issues as needed.
- Expect things will crash -resource will need to plan accordingly and manage their time appropriately
- Data- No managed data in this area
- But if they detect issues that may be a data problem- then go to team that manages data
Required Technologies:
- Azure Cloud
- Python – Code is in Python, need to have a senior level of experience and be able to fix issues
- PySpark- Python interface over Spark;
- Azure Data Factory – Orchestration to oversee the pipeline and trigger the jobs
- Azure DevOps - deployment; If they have GitHub, that works as they can pick up quick
- Databricks: big deal – need strong knowledge, clusters, nodes, Unity catalog
- Model Registry in Unity catalog
- Asure Data Lake Storage (ADLS) – storage
- Data Factory – oversee the pipeline and trigger the pipeline of jobs