Key Responsibilities
- Develop and maintain test automation scripts using PySpark for big data applications.
- Collaborate with data engineers and developers to understand data processing workflows and requirements.
- Design and implement automated tests for data ingestion, processing, and transformation in a Hadoop ecosystem.
- Perform data validation, data integrity, and performance testing for Spark applications.
- Utilize Spark-specific concepts such as RDDs, Data Frames, Datasets, and Spark SQL in test automation.
- Create and manage CI/CD pipelines for automated testing in a big data environment.
- Identify, report, and track defects, and work with the development team to resolve issues.
- Optimize and tune Spark jobs for performance and scalability.
- Maintain and update test cases based on new features and changes in the application.
- Document test plans, test cases, and test results comprehensively.
- Perform QA and manual testing for payments applications, ensuring compliance with business requirements and standards.
- Work with limited direction, usually within a complex environment, to drive delivery of solutions and meet service levels.
- Productively work with stakeholders in multiple countries and time zones.
- With active engagement, collaboration, effective communication, quality, integrity, and reliable delivery develop and maintain a trusted and valued relationship with the team, customers and business partners.
This is a hybrid position. Expectation of days in office will be confirmed by your hiring manager.
Qualifications
Basic Qualifications
Bachelors degree, OR 3+ years of relevant work experience
Preferred Qualifications
bachelor’s degree in computer science, Information Technology or related field.
Relevant certifications in Big Data, Spark, QA.
Experience with on premise infrastructure
Knowledge of ETL processes and tools.