Your responsibilities will include:
- Platform Migration and Modernization: Lead setting up of data in GCP, ensuring data integrity and designing for scalability and performance.
- Pipeline Development and Enhancement: Design, develop, and maintain robust and scalable data pipelines using GCP services (BigQuery, DataFlow, Pub/Sub, etc.). This includes creating reusable components, implementing best practices, and ensuring the pipelines' maintainability and extensibility. Experience with automated data lineage is essential.
- Data Governance and Security: Implement and maintain data governance and security best practices within the GCP environment, ensuring compliance with relevant regulations.
- CI/CD Integration: Integrate data pipelines into our continuous integration and continuous delivery (CI/CD) pipeline.
- Performance Optimization: Continuously monitor and optimize the performance of data pipelines and platforms to ensure fast and reliable data processing.
- Troubleshooting and Debugging: Identify, troubleshoot, and resolve issues related to data pipelines and platforms, providing timely solutions to ensure continuous data flow.
- Reporting and Analysis: Generate reports on data pipeline performance and data quality, analyzing results to identify trends and areas for improvement.
- Mentorship and Collaboration:** Mentor junior team members, sharing your expertise and guiding them in best practices for data engineering. Collaborate closely with developers, data scientists, business analysts, and other stakeholders to ensure alignment on data strategy and execution.
Qualifications
Must Have Skills
- Deep Technical Proficiency: Extensive hands-on experience with GCP services (BigQuery, DataFlow, Pub/Sub, etc.), SQL, and at least one programming language (Python, Java, or similar). Experience with Terraform, Airflow, and other related tools is highly desirable.
- Architectural Design and Scalability: You have a proven ability to design and implement scalable and maintainable data solutions. You understand the challenges of handling large datasets and can design solutions that can meet future growth demands while adhering to strict performance and availability requirements.
- Data Modelling and ETL: You possess strong data modelling skills and experience with ETL processes, including designing and implementing efficient and robust data transformations.
- Collaboration and Communication: You can effectively communicate technical concepts to both technical and non-technical audiences, including senior management. You understand the importance of clear and concise documentation.
Qualifications:
Mandatory
- 5+ years of experience in data engineering, with a focus on data warehousing and ETL development.
- 5+ years of SQL development experience.
- 3+ years of GCP experience with solutions designed and implemented at production scale.
- Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent experience).
- Proven experience in designing and implementing enterprise-level data solutions in a regulated environment.
- Strong understanding of data security, governance, and compliance best practices.
- Excellent analytical and problem-solving skills.
- Excellent communication and collaboration skills.
- Understanding of relevant data privacy and security regulations (e.g., GDPR, CCPA).
Desired
- Experience in Testing – manual or automated
- Experience in FinTech domain