Key Responsibilities:
- Design, develop, and optimize data pipelines using Python, Pyspark, AWS Data Analytics services such as RDS, DMS, Glue, Lambda, Redshift, and Athena.
- Implement data migration and transformation processes using AWS DMS and Glue.
- Work with SQL (Oracle & Postgres) to query, manipulate, and analyse large datasets.
- Develop and maintain ETL/ELT workflows for data ingestion and transformation.
- Utilize AWS services like S3, IAM, CloudWatch, and VPC to ensure secure and efficient data operations.
- Write clean and efficient Python scripts for automation and data processing.
- Collaborate with DevOps teams using Azure DevOps for CI/CD pipelines and infrastructure management.
- Monitor and troubleshoot data workflows to ensure high availability and performance.