Data Engineer - Informatica

Data Engineer - Informatica

Posted 3 days ago by Whitehall Resources

Apply
Negotiable
Inside
Undetermined
London Area, United Kingdom
Apply
Agile Methodology Amazon Elastic Compute Cloud Amazon Web Services (AWS) Apache Spark Artificial Intelligence AWS Elastic MapReduce (EMR) AWS Identity And Access Management (IAM) AWS Lambda Batch Processing Cloud Computing Cloud Technology Code Review Continuous Integration and Continuous Delivery Data Engineering Datamodel Data Modeling Data Quality Extract Transform Load (ETL) Infrastructure as Code (IaC) Management Microsoft Azure Nice (Unix Utility) Performance Tuning Power BI Pyspark Security Clearance SQL (Programming Language) Stakeholder Communications Stakeholder Management Supervision Technical Documentation Terraform Version Control Workflows

Summary: The Senior Data Engineer role focuses on engineering production-grade data pipelines on AWS, utilizing PySpark/Python and SQL, while modernizing legacy workloads and supporting reporting use cases. The position requires collaboration across teams in an Agile environment, with responsibilities including CI/CD practices, mentoring engineers, and ensuring security compliance. The role is based in London and falls inside IR35 regulations. Candidates should have hands-on experience with AWS and Spark, along with a strong background in data engineering at scale.

Key Responsibilities:

  • Engineer production grade data pipelines on AWS (EMR, S3, Lambda), using PySpark/Python and SQL.
  • Migrate and modernise legacy workloads onto cloud native services.
  • Support reporting & MI use cases, including transformations and data models.
  • Own CI/CD and version control practices, review code, and enforce engineering standards.
  • Coach and mentor engineers, provide technical guidance/code reviews.
  • Work in Agile delivery, collaborating across product, data, and platform teams.
  • Embed security and compliance by design, aligning with BPSS/SC constraints.

Key Skills:

  • Hands on expertise in AWS & Spark: Amazon EMR, S3, Lambda; strong PySpark/Python and SQL.
  • Data engineering at scale in government or complex domains.
  • CI/CD & DevOps: pipelines and IaC (e.g., Terraform).
  • Version control & collaboration: Git/GitLab, code review, branching strategies.
  • APIs & integration: building/consuming data services.
  • Agile ways of working with Jira/Confluence; clear stakeholder communication.
  • Security clearance: BPSS (minimum) and SC cleared or SC clearable.

Salary (Rate): undetermined

City: London

Country: United Kingdom

Working Arrangements: undetermined

IR35 Status: inside IR35

Seniority Level: undetermined

Industry: IT

Detailed Description From Employer:

Senior Data Engineer - Informatica

Whitehall Resources currently require an experienced Data Engineer to work with a key client based in London

Please note this role falls INSIDE IR35

What you’ll do

  • Engineer production grade data pipelines on AWS (EMR, S3, Lambda), using PySpark/Python and SQL, with a focus on performance, resilience, testing, and observability.
  • Migrate and modernise legacy workloads (e.g., ETL jobs and reporting feeds) onto cloud native services, creating reusable components and shared frameworks.
  • Support reporting & MI use cases, including transformations and data models that feed downstream tools (e.g., Power BI).
  • Own CI/CD and version control practices (e.g., Git/GitLab), review code, and enforce engineering standards.
  • Coach and mentor engineers, provide technical guidance/code reviews, and contribute to architectural decisions across squads.
  • Work in Agile delivery, collaborating across product, data, and platform teams using Jira/Confluence; translate requirements into robust engineering tasks.
  • Embed security and compliance by design, aligning with BPSS/SC constraints and department data handling policies.

Essential skills & experience

  • Hands on expertise in AWS & Spark: Amazon EMR, S3, Lambda; strong PySpark/Python and SQL for large scale batch processing.
  • Data engineering at scale in government or similarly complex domains, including performance tuning and data quality management.
  • CI/CD & DevOps: pipelines and IaC (e.g., Terraform), automated testing, and release governance.
  • Version control & collaboration: Git/GitLab, code review, branching strategies, and trunk/PR workflows.
  • APIs & integration: building/consuming data services to move and expose data safely and reliably.
  • Agile ways of working with Jira/Confluence; clear stakeholder communication and concise technical documentation.
  • Security clearance: BPSS (minimum) and SC cleared or SC clearable for UK government work.

Desirable

  • Data warehousing & modeling (e.g., Redshift; dimensional modeling; dbt).
  • Basic Power BI familiarity to partner with BI developers and validate end to end data flows.
  • AWS ecosystem depth (Athena, Redshift, EC2, CloudWatch, IAM) and event driven patterns.

Certifications (nice to have)

  • AWS Certified Cloud Practitioner (or higher), Azure AI Fundamentals (awareness of ML/AI services).

SFIA Level 4 (Enable) alignment

  • Autonomy: Works under general direction; plans own work; designs and implements PySpark jobs on EMR, modernising legacy code with minimal supervision.
  • Influence: Shapes standards through code reviews and mentoring; influences delivery outcomes across teams.
  • Complexity: Handles substantial, multifaceted engineering tasks (e.g., migration to new MI platform; data quality resolution; estimating effort).
  • Business skills: Communicates effectively with stakeholders; aligns data products to reporting/decision making needs; contributes to Agile ceremonies.
Apply
Similar Jobs
Loading data...