Data Engineer

Johannesburg, GP, ZA, South Africa

Job Description

About MI-C3





Established in 2014 by CEO and founder Glen Scott, MI-C3 International Limited is a Malta-based company specializing in delivering trusted software solutions tailored for mission-critical environments. Our flagship product, AFFECTLI, empowers organizations to make informed, data-driven decisions by providing a consolidated, real-time view of complex operations. We pride ourselves on fostering a collaborative, agile work environment that celebrates diversity, rewards


Data Integrations Engineer (NiFi)





We are seeking a mid-level Data Integration Engineer with hands-on experience in Apache NiFi to join our dynamic team. In this role, you will design, implement, and maintain real-time data integration pipelines, handling data from diverse sources such as IoT/IIoT devices, third-party APIs, and raw files. Your primary focus will be on processing streaming data to provide valuable insights that drive informed decision-making within our organization.



As MI-C3 transitions towards Fluvio and Rust, experience with these technologies will be advantageous but is not mandatory. The ideal candidate will possess a deep understanding of data pipelines, real-time event streaming, and ETL workflows, coupled with a passion for exploring and implementing new technologies.


Key Responsibilities




Collaborate with cross-functional teams to design and implement scalable, real-time data streaming solutions using Apache NiFi. Ingest and process data from various sources, including IoT/IIoT protocols (e.g., MQTT, SNMP, CoAP, TCP, WebSockets) and third-party APIs. Develop and maintain robust ETL pipelines, ensuring data is transformed and loaded efficiently for analysis and storage. Continuously monitor and optimize data workflows to maintain low-latency, high-throughput processing capabilities. Configure and manage message brokers such as Kafka, RabbitMQ, and AMQP to facilitate efficient data exchange and support event-driven architectures. Implement validation checks and quality measures to ensure the accuracy, reliability, and integrity of integrated data. Proactively identify, diagnose, and resolve issues related to data ingestion, transformation, and streaming processes to ensure uninterrupted data flow.

Technical Requirements




Demonstrated experience in designing and implementing data integration solutions using Apache NiFi for real-time streaming data. Strong skills in Java and Python for developing custom data processing components and applications. Familiarity with tools such as Apache Spark and Kafka for building scalable data integration solutions. Experience configuring and managing message brokers like RabbitMQ, AMQP, and Kafka to enable efficient data exchange. Hands-on experience with protocols such as MQTT, SNMP, CoAP, TCP, and WebSockets for data capture from edge devices and industrial systems. Knowledge of data validation techniques and quality assurance practices to ensure reliable data integration. Strong analytical and problem-solving abilities, with a keen attention to detail. Excellent communication and teamwork skills to effectively collaborate with cross-functional teams. A proactive mindset with a willingness to learn and work with new tools and technologies, including Fluvio and Rust.

Preferred Qualifications




Bachelor's degree in Computer Science, Information Technology, or a related field. Familiarity with Fluvio and Rust is a plusExperience with cloud-based platforms and distributed systems is advantageous. Understanding of embedded systems, requirements engineering, and systems integration is beneficial.

What We Offer




Be part of a forward-thinking company that values innovation and continuous improvement. Opportunities for professional development and career advancement within a growing organization. A supportive and inclusive work environment that values diversity and collaboration. * A comprehensive compensation package commensurate with experience and qualifications.

Beware of fraud agents! do not pay money to get a job

MNCJobs.co.za will not be responsible for any payment made to a third-party. All Terms of Use are applicable.


Related Jobs

Job Detail

  • Job Id
    JD1403378
  • Industry
    Not mentioned
  • Total Positions
    1
  • Job Type:
    Contract
  • Salary:
    Not mentioned
  • Employment Status
    Permanent
  • Job Location
    Johannesburg, GP, ZA, South Africa
  • Education
    Not mentioned