Job Description

A leading organisation is looking for a Data Engineer to join their team. The main purpose of the role is to prepare and manage large volumes of data for analytical or operational uses. This role involves designing, building, and maintaining the architecture used for data processing and storage. This includes tasks such as:

• Data Modeling and Design: Developing data models and structures that are efficient, scalable, and optimized for various data storage and retrieval needs.
• Data Pipeline Construction: Building and maintaining robust data pipelines that ensure the smooth flow of data from various sources to storage systems and data processing applications, this often involves working with ETL (Extract, Transform, Load) processes.
• Database Management: Managing and optimizing databases to ensure they are fast, reliable, and secure. This includes working with SQL and NoSQL databases.
• Big Data Technologies: Utilizing big data technologies like Hadoop, Spark, and Kafka to handle large-scale data processing.
• Automation and Optimization: Automating data processes and optimizing data flow to improve efficiency and reduce the likelihood of errors.
• Ensuring Data Quality and Governance: Implementing measures to ensure data accuracy, consistency, and security, and complying with data privacy laws and regulations.

Responsibilities:

• Provide DataOps support to Analytics Engineers, including overseeing AWS services and CI/CD processes for deploying DDL to a Snowflake Data Warehouse.
• Manage data extraction from various source systems hosted on AWS RDS and EC2 instances
• Maintain and optimize the extract, stage, delta load staging process, ensuring efficient data flow and integrity.
• Transition to and maintain an Operational Data Store (ODS) using Change Data Capture (CDC) for data ingestion across multiple source databases, including MySQL, Postgres, SQL Server, and NoSQL data stores.
• Collaborate with cross-functional teams to understand and meet data requirements, ensuring the data infrastructure supports these needs.
• Ensure data security and compliance with financial industry regulations and best practices
• Troubleshoot and resolve issues related to data systems and processes.
• Continuously evaluate and implement new technologies and tools to enhance data systems and processes.

Requirements:

• Minimum of 3 years of experience in Data Engineering, preferably in a financial services environment.
• Strong experience with cloud data warehousing, including schema change management and DDL deployments. (Snowflake CDW preferable).
• AWS Solutions Architect Associate certification preferable.
• Proven experience in managing AWS services, including RDS and EC2, in a professional setting
• Solid understanding and experience with CI/CD processes and tools, specifically related to data operations.
• Proficiency in SQL and familiarity with programming languages such as Python or Java.

EE Disclaimer: Kindly note that all positions will be filled in accordance with the Company's Employment Equity plan. We also encourage people with disabilities to apply.

Application Unsuccessful Disclaimer: Should you not receive feedback on your application with us within a period of 2 weeks of submission, you may consider your application as being unsuccessful. Please keep an eye on our website and other career sites for future opportunities that may arise.

Afrizan Personnel