Overview
We are seeking experienced Data Engineers to join a major UK Government data programme, building and improving large-scale data solutions.
You will design, develop and maintain reliable ETL/ELT pipelines, working with Python, PySpark, Cloudera Data Platform and Google Cloud Platform (GCP).
This is an opportunity to solve complex data challenges, improve pipeline performance and contribute to modern data engineering practices.
Key Responsibilities
- Design, build and maintain scalable ETL/ELT data pipelines.
- Extract, transform and load data into strategic platforms.
- Profile data, identify quality issues and support data validation.
- Develop reusable Python packages and engineering components.
- Optimise queries, data processing and pipeline performance.
- Improve automation and data engineering processes.
- Work with technical teams and business stakeholders.
- Apply engineering standards, governance and security requirements.
- Evaluate new tools and technologies and share best practice.
Essential Skills & Experience
- Proven experience as a Data Engineer or in a similar role.
- Strong Python programming skills.
- Hands-on ETL/ELT pipeline development.
- Experience with PySpark and large-scale data processing.
- Cloudera Data Platform or comparable enterprise data platforms.
- GCP or equivalent cloud data environments.
- Knowledge of SQL, data profiling and data quality.
- Experience optimising pipelines and query performance.
- Understanding of data governance, security and engineering standards.
- Strong problem-solving and stakeholder collaboration skills.
Technical Skills & Tools
Core Technologies
- Python
- PySpark
- ETL / ELT pipeline development
- Cloudera Data Platform
- Google Cloud Platform (GCP)
Additional Platforms & Capabilities
- SQL and query optimisation
- Large-scale data processing
- Data profiling and quality management
- Data integration and transformation
- Reusable software packages
- Cloud data platforms
- Pipeline monitoring and automation
- CI/CD and version control
- Data governance and security
Experience across every listed technology is not essential. Strong transferable data engineering experience is welcomed.
SFIA Level Expectations
- Independently designs and delivers data engineering solutions.
- Applies sound technical judgement to complex challenges.
- Builds scalable, reusable and maintainable pipelines.
- Identifies and resolves data quality and performance issues.
- Recommends improvements to tools and processes.
- Works effectively with technical teams and stakeholders.
- Shares knowledge and promotes engineering best practice.
Desirable
- UK Government or public-sector experience.
- Experience with large-scale or complex datasets.
- Strong Cloudera Data Platform experience.
- GCP data engineering experience.
- Development of reusable Python packages.
- Experience improving enterprise data pipelines.
- Knowledge of data governance and compliance.
- Contribution to engineering standards or communities.
Clearance Requirement
SC Security Clearance is required for this assignment.
Candidates must be able to confirm their current clearance status and meet the required screening and onboarding requirements.