Job Description

What you will be doing:
We are looking for an experienced Data Engineer with strong Python and SQL skills, as well as hands-on experience with Kafka, Docker, and Kubernetes. The ideal candidate will have a proven track record in building bespoke data processing pipelines, developing microservices, and designing data table structures and schemas. You will collaborate with cross-functional teams to integrate and optimize data systems, while working with both structured and unstructured data.Key Responsibilities:

• Design and Build Data Pipelines: Develop and maintain custom, scalable data pipelines for processing large datasets, including unstructured text and document data.
• Integrate Data Systems: Collaborate with diverse teams to understand data sources and build robust integrations with various internal and external systems.
• Build Microservices: Develop microservices to handle data processing tasks, ensuring they are scalable, secure, and efficient.
• Optimize Performance: Design and optimize table structures, schemas, and database queries for improved data retrieval and storage performance.
• Kafka and Real-time Data: Leverage Kafka for real-time data streaming and processing, ensuring high availability and low-latency data delivery.
• Containerization & Orchestration: Use Docker and Kubernetes for deploying and managing data applications in production environments.
• Collaborate with Teams: Work closely with data scientists, analysts, and other engineers to define requirements and deliver high-quality, reliable data solutions.
• Ensure Data Quality: Maintain data integrity and implement best practices to ensure the consistency and quality of the data being processed

What we are looking for:

• Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent practical experience).
• 5 years experience as Date Engineer
• Strong proficiency in Python: Experience in developing scalable data applications, automating processes, and implementing algorithms.
• Strong SQL skills: Ability to design efficient queries and optimize database performance.
• Hands-on experience with Kafka, including setting up and managing streams, and integrating with data pipelines.
• Proficiency with Docker and Kubernetes for containerization and orchestration in production environments.
• Experience building bespoke data processing pipelines: Expertise in handling large volumes of data from multiple sources and optimizing processing workflows.
• Experience building microservices to deliver modular, scalable, and maintainable data solutions.
• Experience designing table structures and schemas that meet both performance and business requirements.
• Ability to collaborate with diverse teams, understanding their needs and building integrations that allow for efficient data flow.
• Java experience is advantageous, particularly for implementing additional data processing logic or integration points.
• Experience with unstructured text and document data is a plus, especially in processing and transforming such data for analytical purposes.

Please note that if you do not hear from us within 3 weeks, consider your application unsuccessful.Please note that most of our positions are remote however candidates should be residing within the traveling distance as circumstance of the opportunity can change.

ExecutivePlacements.com