Role and responsibilities
As a Data Engineer at IBM, you'll play a vital role in the development, design of application, provide regular support/guidance to project teams on complex coding, issue resolution and execution.
- Your primary responsibilities include:
- Lead the design and construction of new solutions using the latest technologies, always looking to add business value and meet user requirements.
- Strive for continuous improvements by testing the build solution and working under an agile framework.
- Discover and implement the latest technologies trends to maximize and build creative solutions
Required education
Bachelor's Degree
Preferred education
Master's Degree
Required technical and professional expertise
- Experience with Apache Spark (PySpark): In-depth knowledge of Spark’s architecture, core APIs, and PySpark for distributed data processing.
- Big Data Technologies: Familiarity with Hadoop, HDFS, Kafka, and other big data tools. Data Engineering Skills: Strong understanding of ETL pipelines, data modeling, and data warehousing concepts.
- Strong proficiency in Python: Expertise in Python programming with a focus on data processing and manipulation. Data Processing Frameworks: Knowledge of data processing libraries such as Pandas, NumPy.
- SQL Proficiency: Experience writing optimized SQL queries for large-scale data analysis and transformation.
- Cloud Platforms: Experience working with cloud platforms like AWS, Azure, or GCP, including using cloud storage systems
Preferred technical and professional experience
- Define, drive, and implement an architecture strategy and standards for end-to-end monitoring.
- Partner with the rest of the technology teams including application development, enterprise architecture, testing services, network engineering,
- Good to have detection and prevention tools for Company products and Platform and customer-facing