Senior Data Engineer (Azure/DataBricks)
Posted Today by GIOS Technology
£61 Per hour
Undetermined
Onsite
Manchester Area, United Kingdom
Summary: The Lead Azure Databricks Engineer role in Manchester involves driving the design, build, optimization, and governance of enterprise data platforms. The position requires a blend of technical expertise and leadership, particularly within the Lloyd’s of London market, to shape cloud data strategy and ensure robust solutions. This hands-on role emphasizes collaboration with various functions to meet regulatory requirements and enhance platform performance.
Key Responsibilities:
- Lead the development and optimization of large-scale data platforms using Azure and Databricks.
- Work closely with various functions to deliver solutions aligned to Lloyd’s of London regulatory and operational requirements.
- Partner with architects, SMEs, product owners, and leadership to translate business needs into scalable cloud data solutions.
- Own and resolve complex technical challenges with speed and clarity.
- Champion best-practice design patterns, data lifecycle management, CI/CD, automation, and cloud-native engineering principles.
- Provide technical leadership and mentoring, setting engineering standards and uplifting capability across the team.
- Drive continuous improvement, proactively identifying opportunities to enhance platform performance, reduce cost, and improve resilience.
- Oversee end-to-end data engineering delivery, ensuring high-quality pipelines, reliability, observability, and performance.
Key Skills:
- Expert-level engineering experience across Azure Data Services and Databricks in enterprise-scale environments.
- Deep understanding of Delta Lake, Medallion architecture, distributed compute, Lakehouse patterns, data modelling and optimisation.
- Strong Python, PySpark and Spark SQL skills, plus experience implementing CI/CD for data solutions.
- Solid understanding of data governance, lineage, access controls, FinOps, and secure cloud engineering practices.
- Outstanding communication skills with the ability to interface confidently with senior stakeholders.
- Delivery-focused mindset with a proven track record of ownership and accountability.
- Experience working in fast-paced, high-expectation environments with tight deadlines and evolving requirements.
Salary (Rate): £61.00/hr
City: Manchester
Country: United Kingdom
Working Arrangements: on-site
IR35 Status: undetermined
Seniority Level: undetermined
Industry: IT
We are looking for Lead Azure Databricks Engineer at Manchester, UK – 2 days per week Onsite Role Description: We are seeking an exceptional Lead Azure & Databricks Engineer to drive the design, build, optimisation, and governance of our enterprise data platforms. The ideal candidate combines deep technical expertise with hands-on leadership, excels in complex and highly regulated environments, and has proven delivery experience within the Lloyd’s of London market. You will play a pivotal role in shaping our cloud data strategy, enabling advanced analytics, and ensuring robust, scalable and secure solutions across our organisation. This is 100% hands on role.
Key Responsibilities
- Lead the development, and optimisation of large-scale data platforms using Azure (ADF, Data Lake, Key Vault, Azure Functions) and Databricks (Delta Lake, PySpark, Unity Catalog).
- Work closely with underwriting, actuarial, delegated authority, bordereaux, exposure management, Reinsurance, Finance Solvency II and risk functions to deliver solutions aligned to Lloyd’s of London regulatory and operational requirements, including compliance, auditability, and data lineage.
- Partner with architects, SMEs, product owners, and leadership to translate business needs into scalable cloud data solutions.
- Own and resolve complex technical challenges with speed and clarity—someone who consistently gets the job done.
- Champion best-practice design patterns, data lifecycle management, CI/CD, automation, and cloud-native engineering principles.
- Provide technical leadership and mentoring, setting engineering standards and uplifting capability across the team.
- Drive continuous improvement, proactively identifying opportunities to enhance platform performance, reduce cost, and improve resilience.
- Oversee end-to-end data engineering delivery, ensuring high-quality pipelines, reliability, observability, and performance.
Required Experience & Skills
- Expert-level engineering experience across Azure Data Services and Databricks in enterprise-scale environments.
- Deep understanding of Delta Lake, Medallion architecture, distributed compute, Lakehouse patterns, data modelling and optimisation.
- Strong Python, PySpark and Spark SQL skills, plus experience implementing CI/CD for data solutions (Azure DevOps, Branching and merging).
- Solid understanding of data governance, lineage, access controls, FinOps, and secure cloud engineering practices.
- Outstanding communication skills—able to interface confidently with senior stakeholders, articulate complex concepts clearly, and lead cross-functional discussions.
- Delivery-focused mindset with a proven track record of ownership, accountability, and seeing initiatives through to completion.
- Experience working in fast-paced, high-expectation environments with tight deadlines and evolving requirements.