- Design, develop, maintain efficient and scalable solutions using PySpark
- Ensure data quality and integrity by implementing robust testing, validation and cleansing processes
- Integrate data from various sources, including databases, APIs, external datasets etc.
- Optimize and tune PySpark jobs for performance and reliability
- Document data engineering processes, workflows and best practices
- Strong understanding of databases, data modelling, and ETL tools and processes
- String programming skills in python and proficiency with PySpark, SQL
- Experience with relational databases, Hadoop, Spark, Hive, AWS
- Excellent communication and collaboration skills
Skill
pyspark
Data Engineer
Minimum Qualification
Any Graduate