Senior Data Engineer

Pay

On application

Type

Contract

Experience

5–10 years

Level

Senior

Sector

Agriculture, Forestry & Fishing

Closes

in 30 days

Recruiter

Mediro ICT · All jobs at Mediro ICT

About the role

Our client in Fourways is looking for a Senior Data Engineer to join their team. The Data Engineer will provide hands-on technical delivery and optimisation support for the Digital Agriculture Agri Data Hub, developed on Azure Databricks. The platform integrates structured data from multiple on-premises databases and unstructured SharePoint content to support business intelligence, agronomic analytics and AI-driven capabilities. The project has progressed into User Acceptance Testing (UAT), and the resource will focus on resolving defects, stabilising data pipelines, optimising performance and supporting production readiness.

The role requires a technically strong, delivery-focused engineer who can operate independently within an established architecture and contribute to ongoing MVP improvements.

• The Data Engineer will join an existing Digital Agriculture implementation focused on delivering a centralised Agri Data Hub within Azure Databricks. The solution incorporates structured data ingestion from on-premises databases into a Medallion Architecture, curated Gold-layer data products supporting BI reporting, and a Generative AI/LLM knowledge agent (Agripedia) consuming unstructured SharePoint information.
• Current project position: Analysis and design, core architecture, Azure networking, CI/CD pipelines, Gold-layer data modelling and initial BI and LLM development have been completed. The BI reports and Agripedia solution are currently in UAT, with defect resolution and production readiness activities underway.
• Technical delivery and stabilisation: Review and understand the existing Databricks implementation, ingestion pipelines, data transformations and Gold-layer models. Investigate and resolve defects identified during UAT, troubleshoot integration and data-quality issues, and implement improvements required to stabilise the solution.
• Pipeline development and optimisation: Develop, maintain and optimise ingestion pipelines for structured data from on-premises databases. Implement scalable transformation logic using Spark, PySpark and SQL, ensuring that pipelines are performant, secure, cost-efficient and supportable. Apply Delta Lake optimisation techniques and maintain consistency across the established Medallion Architecture.
• Production readiness and deployment: Support the transition from UAT into production, including validation of data accuracy, pipeline execution, error handling, monitoring and recovery processes. Work within existing Azure DevOps and CI/CD deployment processes, ensuring changes are appropriately tested and deployed.
• Cost, performance and operational improvements: Monitor and optimise Databricks compute usage, pipeline performance and associated cloud costs as the solution moves into production. Identify opportunities to improve operational efficiency, stability and maintainability.
• AI and unstructured data integration: Support the existing Agripedia LLM knowledge-agent solution where data engineering input is required, particularly relating to SharePoint data ingestion, integration, data preparation and ongoing optimisation. Previous experience supporting AI/LLM data pipelines would be advantageous.
• Governance, documentation and knowledge transfer: Adhere to existing enterprise governance and architecture standards, contribute to data-quality controls, and update technical documentation where required. Collaborate with the Data Architect and internal technical teams to support knowledge transfer, troubleshooting and further enhancements.
• Ongoing MVP enhancements: As the solution is being delivered iteratively, support further development and refinement of data pipelines, curated data products and platform capabilities based on business priorities, testing feedback and production requirements.

What they're looking for

• Databricks, Microsoft Azure, Spark, PySpark, SQL, Delta Lake, Medallion Architecture (Bronze/Silver/Gold), Azure DevOps, CI/CD, Data Integration, Data Quality, Performance and Cost Optimisation. Exposure to Generative AI/LLM solutions and SharePoint integration would be advantageous.
• Strong intermediate to senior-level engineer with proven hands-on Databricks experience in production environments. Must demonstrate strong troubleshooting and problem-solving capabilities, with the ability to understand existing solutions, identify root causes and implement practical improvements. Comfortable working within an established architecture and governed enterprise environment, collaborating with architects, technical teams and business stakeholders. Must be proactive, adaptable and capable of taking ownership of technical activities with minimal supervision. Experience supporting UAT, production deployments and iterative MVP delivery is important.

More IT and developer jobs · More IT and developer jobs in Sandton · All jobs at Mediro ICT