AWS Spark Python PySpark Data Engineer - MUST HAVE SC CLEARANCE - Remote - 6 months+
Posted Today by Octopus Computer Associates
£437 Per day
Inside
Remote
Remote, UK
Summary: The role of AWS Spark Python PySpark Data Engineer requires SC clearance and involves designing, building, and operating data solutions for critical analytics in a public sector environment. The position focuses on engineering production-grade data pipelines on AWS, modernizing legacy workloads, and mentoring other engineers. The role is primarily remote, with occasional meetings in Newcastle or Leeds. Candidates must have hands-on expertise in AWS and Spark technologies, along with strong PySpark and SQL skills.
Key Responsibilities:
- Engineer production grade data pipelines on AWS (EMR, S3, Lambda) using PySpark/Python and SQL.
- Migrate and modernise Legacy workloads onto cloud native services.
- Support reporting & MI use cases, including transformations and data models.
- Own CI/CD and version control practices, review code, and enforce engineering standards.
- Coach and mentor engineers, provide technical guidance/code reviews.
- Work in Agile delivery, collaborating across product, data, and platform teams.
- Embed security and compliance by design, aligning with BPSS/SC constraints.
Key Skills:
- Hands-on expertise in AWS & Spark: Amazon EMR, S3, Lambda; strong PySpark/Python and SQL.
- Data engineering at scale in government or complex domains.
- CI/CD & DevOps: pipelines and IaC (eg, Terraform), automated testing.
- Version control & collaboration: Git/GitLab, code review, branching strategies.
- APIs & integration: building/consuming data services.
- Agile ways of working with Jira/Confluence; clear stakeholder communication.
- Security clearance: BPSS (minimum) and SC cleared or SC clearable.
Salary (Rate): £437 per day
City: undetermined
Country: UK
Working Arrangements: remote
IR35 Status: inside IR35
Seniority Level: Senior
Industry: IT
AWS Spark Python PySpark Data Engineer - MUST HAVE SC CLEARANCE - Remote - 6 months+/RATE: £437 per day inside IR35
One of our Blue Chip Clients is urgently looking for a AWS Spark Python PySpark Data Engineer.
For this role you can work remotely.
Please find some details below:
Work type: this role is remote, however there may be an infrequent ask to attend a meeting in Newcastle or Leeds.
Job Description:
We're seeking an SC cleared Senior Data Engineer who have Informatica skill to design, build, and operate data solutions that power mission critical analytics in a complex public sector environment. You'll lead on scalable pipelines (PySpark on Amazon EMR), modernise Legacy estates, and mentor engineers-turning raw data into reliable, secure, and actionable intelligence for stakeholders.
What you'll do
Engineer production grade data pipelines on AWS (EMR, S3, Lambda), using PySpark/Python and SQL, with a focus on performance, resilience, testing, and observability.
Migrate and modernise Legacy workloads (eg, ETL jobs and reporting feeds) onto cloud native services, creating reusable components and shared frameworks.
Support reporting & MI use cases, including transformations and data models that feed downstream tools (eg, Power BI).
Own CI/CD and version control practices (eg, Git/GitLab), review code, and enforce engineering standards.
Coach and mentor engineers, provide technical guidance/code reviews, and contribute to architectural decisions across squads.
Work in Agile delivery, collaborating across product, data, and platform teams using Jira/Confluence; translate requirements into robust engineering tasks.
Embed security and compliance by design, aligning with BPSS/SC constraints and department data handling policies.
Essential skills & experience
Hands on expertise in AWS & Spark: Amazon EMR, S3, Lambda; strong PySpark/Python and SQL for large scale batch processing.
Data engineering at scale in government or similarly complex domains, including performance tuning and data quality management.
CI/CD & DevOps: pipelines and IaC (eg, Terraform), automated testing, and release governance.
Version control & collaboration: Git/GitLab, code review, branching strategies, and trunk/PR workflows.
APIs & integration: building/consuming data services to move and expose data safely and reliably.
Agile ways of working with Jira/Confluence; clear stakeholder communication and concise technical documentation.
Security clearance: BPSS (minimum) and SC cleared or SC clearable for UK government work.
Desirable
Data warehousing & modeling (eg, Redshift; dimensional modeling; dbt).
Basic Power BI familiarity to partner with BI developers and validate end to end data flows.
AWS ecosystem depth (Athena, Redshift, EC2, CloudWatch, IAM) and event driven patterns.
Certifications (nice to have)
AWS Certified Cloud Practitioner (or higher), Azure AI Fundamentals (awareness of ML/AI services).
SFIA Level 4 (Enable) alignment
Autonomy: Works under general direction; plans own work; designs and implements PySpark jobs on EMR, modernising Legacy code with minimal supervision.
Influence: Shapes standards through code reviews and mentoring; influences delivery outcomes across teams.
Complexity: Handles substantial, multifaceted engineering tasks (eg, migration to new MI platform; data quality resolution; estimating effort).
Business skills: Communicates effectively with stakeholders; aligns data products to reporting/decision making needs; contributes to Agile ceremonies.
Please send CV for full details and immediate interviews. We are a preferred supplier to the client.
Aleksandra Pytlak-Ratajczyk