Data Engineer (Python)

Summary: The role of a Python Data Engineer involves designing and deploying scalable data processing solutions with a focus on maintainable and test-driven code. The position requires strong hands-on experience in Behave-based unit testing, PySpark development, Delta Lake optimisation, and Azure cloud services. The engineer will be responsible for developing data pipelines and integrating various Azure services within a containerised environment. This role is ideal for candidates who thrive in fast-paced environments and have a solid background in data engineering best practices.

Key Responsibilities:

• Develop and maintain data ingestion, transformation, and validation pipelines using Python and PySpark.
• Implement unit and behavior-driven testing with Behave, ensuring robust mocking and patching of dependencies.
• Design and maintain Delta Lake tables for optimised query performance, ACID compliance, and incremental data loads.
• Build and manage containerised environments using Docker for consistent development, testing, and deployment.
• Develop configurable, parameter-driven codebases to support modular and reusable data solutions.
• Integrate Azure services, including:
  • Azure Functions for serverless transformation logic
  • Azure Key Vault for secure credential management
  • Azure Blob Storage for data lake operations

Key Skills:

• Proven experience in Python, PySpark, and Delta Lake.
• SC Cleared
• Strong knowledge of Behave for test-driven development.
• Experience with Docker and containerised deployments.
• Familiarity with Azure cloud services and data engineering best practices.
• Ability to deliver scalable, maintainable, and testable solutions in a fast-paced environment.

Salary (Rate): £450.0 per day

City: London

Country: UK

Working Arrangements: undetermined

IR35 Status: undetermined

Seniority Level: undetermined

Industry: IT

Detailed Description From Employer:

About the RoleWe are looking for a Python Data Engineer with strong hands-on experience in Behave-based unit testing, PySpark development, Delta Lake optimisation, and Azure cloud services. This role focusses on designing and deploying scalable data processing solutions in a containerised environment, emphasising maintainable, configurable, and test-driven code delivery.
Key Responsibilities

• Develop and maintain data ingestion, transformation, and validation pipelines using Python and PySpark.
• Implement unit and behavior-driven testing with Behave, ensuring robust mocking and patching of dependencies.
• Design and maintain Delta Lake tables for optimised query performance, ACID compliance, and incremental data loads.
• Build and manage containerised environments using Docker for consistent development, testing, and deployment.
• Develop configurable, parameter-driven codebases to support modular and reusable data solutions.
• Integrate Azure services, including:
  • Azure Functions for serverless transformation logic
  • Azure Key Vault for secure credential management
  • Azure Blob Storage for data lake operations

What We're Looking For

• Proven experience in Python, PySpark, and Delta Lake.
• SC Cleared
• Strong knowledge of Behave for test-driven development.
• Experience with Docker and containerised deployments.
• Familiarity with Azure cloud services and data engineering best practices.
• Ability to deliver scalable, maintainable, and testable solutions in a fast-paced environment.

If you're interested in this role, click 'apply now' to forward an up-to-date copy of your CV, or call us now.

Hays Specialist Recruitment Limited acts as an employment agency for permanent recruitment and employment business for the supply of temporary workers. By applying for this job you accept the T&C's, Privacy Policy and Disclaimers which can be found on our website.