Python / Spark / AI Developer

Hire Resolve connects Job seekers to Employers. The company is a Specialist Recruitment & Staffing company with a national presence with a customer base of listed companies on the JSE and global multinationals. The company prides itself on its technical & industry knowledge. The consultants at Hire Resolve are very well networked within their community - they are industry savvy & technically competent. Specialties Specialist Recruitment, Engineering, Mining, Civil Engineering, Finance, Information Technology, C#, Java, PHP, Construction & Building, SAP, Supply Chain, Freight & Logistics, FMCG, Trade & Technical, and Human Resources . Job Description We are looking for a skilled Python / Spark / AI Developer to join a growing technology team. You will be responsible for developing scalable data pipelines, replatforming legacy SQL workloads onto modern Spark and lakehouse technologies, and supporting the development of APIs and AI-driven features. The ideal candidate is a strong Python developer with hands-on production experience in Apache Spark/PySpark, SQL, Delta Lake and REST APIs. Key Responsibilities Build Spark/PySpark data pipelines using Delta Lake. Replatform legacy T-SQL logic into Spark SQL. Develop modern, type-safe and well-tested Python applications. Build REST APIs using FastAPI or equivalent frameworks. Implement authentication, idempotency and job-status functionality. Validate migrated pipelines against legacy systems and provide parity evidence. Work from technical specifications, design documents and ADRs. Develop automated tests using pytest. Contribute to CI/CD pipelines, code reviews and technical documentation. Minimum Requirements 4+ years of professional Python development experience. 2+ years of production experience with Apache Spark / PySpark. Strong knowledge of Spark SQL and DataFrame APIs. Production experience with Delta Lake or equivalent technologies such as Iceberg or Hudi. Strong SQL skills and experience working with legacy T-SQL. Experience with pytest and automated testing. Experience developing REST APIs, preferably with FastAPI. Experience with Docker and containerised development environments. Strong Git and CI/CD experience. Experience with type-hinted Python and static analysis tools such as Pyright or Mypy. Experience with modern Python tooling such as uv or Poetry. Ability to work independently from written technical requirements and design documentation. Bachelor's degree in Computer Science, Engineering or a related field, or equivalent experience. Proven track record of delivering production Python/Spark pipelines independently. Advantageous Skills Experience with LLM / AI integration. Experience with Ollama, vLLM, llama.cpp, LM Studio or hosted AI APIs. Knowledge of data governance, privacy engineering and POPIA/GDPR principles. Experience with Hive Metastore, Trino or Apache Ranger. Azure experience, including ADLS Gen2, Synapse, ADF, Key Vault or Service Bus. Experience with structured logging, OpenTelemetry or OpenLineage. Data migration and pipeline parity testing experience. Databricks Certified Developer for Apache Spark or equivalent certification.