Data Engineer

We are sourcing for a Data Engineer role.

Duties and Responsibilities

Contribute to the establishment and operation of the AI Centre of Excellence (AI CoE)

• Collaborate with executives to develop strategies that align with organisational imperatives.
• Partner with AI Champions, ICT, Data Scientists and AI/ML Engineers to integrate AI into operational workflows smoothly.

Data Ingestion & Integration

• Build scalable batch and streaming pipelines from internal and external sources (APIs, CDC, files, events).
• Implement schema evolution, late-arriving data handling, and idempotent ingestion.
• Set up connectors and jobs across cloud services, message queues (Kafka/Kinesis/Event Hubs), and integration tools.

Data Transformation & Modeling

• Design and maintain curated datasets (bronze/silver/gold) using ELT/ETL.
• Implement dimensional models, data vault, and/or lakehouse patterns as appropriate.
• Use transformation frameworks (e.g., dbt, Spark) with robust testing and documentation.
• Data Platform Engineering
• Build and operate data lakes / warehouses / lake houses (e.g., Delta Lake, BigQuery, Synapse).
• Manage storage formats and optimizations (Parquet/ORC, partitioning, compaction, etc).
• Implement cataloguing, lineage, and discovery (e.g., Unity Catalog, Purview, Data Catalogue).

Reliability, Observability & Quality

• Implement data testing (unit, expectations), monitoring (freshness, volume, schema, distribution), and alerting (e.g., Great Expectations, Soda, Monte Carlo).
• Establish SLAs/SLOs and incident response for data pipelines.
• Ensure reproducibility, versioning, and backfills with orchestration (e.g., Airflow, Dagster, Prefect).

Security, Governance & Compliance

• Apply data access controls (RBAC/ABAC), masking, tokenization, encryption at rest/in transit.
• Enforce data retention, lineage, and regulatory compliance (POPIA or industry-specific).
• Collaborate with Data Governance and Security teams on policies and audits.

Cost & Performance Optimization (FinOps)

• Monitor compute/storage spend and optimize partitioning, caching, spot/preemptible usage.
• Right-size warehouses/clusters and tune jobs for performance and cost efficiency.

Collaboration & Enablement

• Partner with Data Scientists/Analysts on feature engineering, model-ready tables, and MLOps handoffs.
• Create self-service patterns, templates, and documentation to accelerate downstream teams.
• Contribute to platform standards, reusable components, and developer experience.
• Provide mentorship to emerging data engineers

Education, Skills and Experience

• A diploma or bachelor’s degree in computer science, Information Systems, Information Technology, Engineering, or a related field (or equivalent experience) is required.
• 6+ years in Data Engineering or similar roles (Software Engineer with data focus, ETL Developer).
• Experience with Lakehouse platforms (Databricks, Snowflake, BigQuery, Synapse/OneLake).
• Hands-on with Terraform/Bicep/CloudFormation for infrastructure-as-code.
• Familiarity with MLOps (feature stores, model registries, batch/stream scoring integration).
• Proven experience building and operating production-grade data pipelines

Tebogo Rankhumise