Experience : 3 to 7 Years
Location : Mumbai/Bangalore/Trivandrum (Hybrid)
Key Responsibilities:
- Design & Architecture: Collaborate with cross-functional teams to design CI/CD target architecture to automate and integrate Apache Flink with existing data systems.
- Data Streaming Integration: Implement Apache Flink for real-time data processing and ensure seamless integration with the Amazon Managed Service for Apache Flink.
- Migration Projects: Lead efforts to migrate existing Dataflow (Apache Beam) jobs to Apache Flink, ensuring minimal disruption and optimal performance.
- Data Publishing: Develop solutions to publish streaming data from Amazon MSK into the bank's Data Platform using Apache Flink.
- Fault Tolerance Implementation: Implement fault tolerance mechanisms and automated retries in Apache Flink applications as per platform requirements.
- Performance Optimization: Monitor the health and performance of Flink applications, optimizing them for scalability and reliability.
Required Skills & Qualifications:
- Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a related field.
- Proven experience as a Data Engineer with a strong focus on stream processing using Apache Flink.
- Proficiency in programming languages such as Java, Scala, or Python (PyFlink).
- Deep understanding of distributed data processing concepts and frameworks.
- Experience with AWS services, particularly Amazon Managed Service for Apache Flink and Amazon MSK.
- Familiarity with CI/CD practices and tools for automation.
- Strong problem-solving skills and ability to work collaboratively in a team environment.
- Excellent communication skills to effectively collaborate with technical and non-technical stakeholders.
Preferred Qualifications:
- Experience with other stream processing technologies such as Apache Kafka, Apache Beam, or Spark Streaming.
- Knowledge of containerization technologies like Docker and Kubernetes.
- Familiarity with data modeling principles and SQL databases.