Senior Data Engineer – Informatica
Posted 2 days ago by Whitehall Resources
Negotiable
Inside
Undetermined
London
Summary: The Senior Data Engineer role focuses on engineering production-grade data pipelines using AWS technologies, particularly PySpark/Python and SQL. The position involves migrating legacy workloads to cloud-native services and supporting reporting use cases while ensuring compliance with security standards. The candidate will also mentor junior engineers and collaborate in an Agile environment. This role is classified as inside IR35.
Key Responsibilities:
- Engineer production grade data pipelines on AWS (EMR, S3, Lambda), using PySpark/Python and SQL.
- Migrate and modernise legacy workloads onto cloud native services.
- Support reporting & MI use cases, including transformations and data models.
- Own CI/CD and version control practices, review code, and enforce engineering standards.
- Coach and mentor engineers, provide technical guidance/code reviews.
- Work in Agile delivery, collaborating across product, data, and platform teams.
- Embed security and compliance by design, aligning with BPSS/SC constraints.
Key Skills:
- Hands on expertise in AWS & Spark: Amazon EMR, S3, Lambda; strong PySpark/Python and SQL.
- Data engineering at scale in government or similarly complex domains.
- CI/CD & DevOps: pipelines and IaC (e.g., Terraform), automated testing.
- Version control & collaboration: Git/GitLab, code review, branching strategies.
- APIs & integration: building/consuming data services.
- Agile ways of working with Jira/Confluence; clear stakeholder communication.
- Security clearance: BPSS (minimum) and SC cleared or SC clearable.
Salary (Rate): undetermined
City: London
Country: United Kingdom
Working Arrangements: undetermined
IR35 Status: inside IR35
Seniority Level: undetermined
Industry: IT
Senior Data Engineer – Informatica
Whitehall Resources currently require an experienced Data Engineer to work with a key client based in London
**Please note this role falls INSIDE IR35**
What you’ll do
* Engineer production grade data pipelines on AWS (EMR, S3, Lambda), using PySpark/Python and SQL, with a focus on performance, resilience, testing, and observability.
* Migrate and modernise legacy workloads (e.g., ETL jobs and reporting feeds) onto cloud native services, creating reusable components and shared frameworks.
* Support reporting & MI use cases, including transformations and data models that feed downstream tools (e.g., Power BI).
* Own CI/CD and version control practices (e.g., Git/GitLab), review code, and enforce engineering standards.
* Coach and mentor engineers, provide technical guidance/code reviews, and contribute to architectural decisions across squads.
* Work in Agile delivery, collaborating across product, data, and platform teams using Jira/Confluence; translate requirements into robust engineering tasks.
* Embed security and compliance by design, aligning with BPSS/SC constraints and department data handling policies.
Essential skills & experience
* Hands on expertise in AWS & Spark: Amazon EMR, S3, Lambda; strong PySpark/Python and SQL for large scale batch processing.
* Data engineering at scale in government or similarly complex domains, including performance tuning and data quality management.
* CI/CD & DevOps: pipelines and IaC (e.g., Terraform), automated testing, and release governance.
* Version control & collaboration: Git/GitLab, code review, branching strategies, and trunk/PR workflows.
* APIs & integration: building/consuming data services to move and expose data safely and reliably.
* Agile ways of working with Jira/Confluence; clear stakeholder communication and concise technical documentation.
* Security clearance: BPSS (minimum) and SC cleared or SC clearable for UK government work.
Desirable
* Data warehousing & modeling (e.g., Redshift; dimensional modeling; dbt).
* Basic Power BI familiarity to partner with BI developers and validate end to end data flows.
* AWS ecosystem depth (Athena, Redshift, EC2, CloudWatch, IAM) and event driven patterns.
Certifications (nice to have)
* AWS Certified Cloud Practitioner (or higher), Azure AI Fundamentals (awareness of ML/AI services).
SFIA Level 4 (Enable) alignment
* Autonomy: Works under general direction; plans own work; designs and implements PySpark jobs on EMR, modernising legacy code with minimal supervision.
* Influence: Shapes standards through code reviews and mentoring; influences delivery outcomes across teams.
* Complexity: Handles substantial, multifaceted engineering tasks (e.g., migration to new MI platform; data quality resolution; estimating effort).
* Business skills: Communicates effectively with stakeholders; aligns data products to reporting/decision making needs; contributes to Agile ceremonies.
All of our opportunities require that applicants are eligible to work in the specified country/location, unless otherwise stated in the job description.
Whitehall Resources are an equal opportunities employer who value a diverse and inclusive working environment. All qualified applicants will receive consideration for employment without regard to race, religion, gender identity or expression, sexual orientation, national origin, pregnancy, disability, age, veteran status, or other characteristics.