What you need to know in 60 seconds
Data engineering is the work that turns raw records from many systems into data a public body can rely on: collected on time, checked, joined up, documented and served to the people and services that need it. Citizens never see it, but they feel it whenever a payment, a decision or a published statistic turns out to be wrong.
Government is not short of data. It is short of data it can trust across systems. In the State of Digital Government Review, only 27% of survey respondents said their data infrastructure gives a comprehensive view of operations or transactions, and 70% said their data was not well co-ordinated or interoperable and gave them no unified source of truth[1].
This article walks through the six stages of a government data pipeline in plain English, what goes wrong when a stage is skipped, and how to show a minister, an auditor or a member of the public that the numbers hold up.
What is data engineering, in plain English?
Data engineering is building and running the routes that carry data from where it is created to where it is used, along with the checks that make sure it arrives intact. A caseworker types an address into one system. The payments team needs it in another. An analyst needs it, anonymised, in a dashboard. Data engineering is everything in between.
It is not the same as data science. Data scientists ask questions of data. Data engineers make sure there is data worth asking questions of, in a shape that can be trusted, every day and not only on the day of the demo. Government recognises the difference: data engineer is a role of its own in the DDaT capability framework, separate from data scientist and data analyst, as our guide to DDaT roles sets out.
Why does trust in a digital public service depend on it?
Because a service is only as reliable as the data under it. The Government Data Quality Framework puts it plainly: poor or unknown quality data weakens evidence, undermines trust, and ultimately leads to poor outcomes[2]. In a public service those outcomes are concrete: a benefit paid late, a vulnerable person missed, a parliamentary answer that has to be corrected.
Most of that data still lives in old systems. The same review estimated that legacy technology made up 28% of central government systems in 2024[1]. Good pipelines are how a department gets trusted data out of those systems now, whether the source is a case management system, a mainframe or a legacy data warehouse, instead of waiting years for a replacement platform.
What happens at each stage of a government data pipeline?
Public sector data platforms differ in technology, but every pipeline on them does the same six jobs. Skipping one doesn't remove the work. It moves it somewhere more expensive.
| Stage | What it does, in plain English | What goes wrong without it |
|---|---|---|
| Ingestion | Collects data from source systems on a schedule or as it changes, without disturbing them | Manual extracts, emailed spreadsheets, and nobody sure which copy is current |
| Transformation | Cleans and reshapes data into agreed definitions, so "case", "claimant" and "closed" mean one thing | Two reports, two numbers, one awkward meeting |
| Quality checks | Tests every load for gaps, duplicates, bad formats and values that contradict each other | Errors found by the public instead of by the department |
| Modelling and storage | Holds data in a governed store with history, so you can see what changed and when | Last month's figure can't be reproduced |
| APIs | Serves data to other systems and services in a controlled, documented way | Point-to-point links nobody can maintain or switch off |
| Reporting and analytics | Turns trusted data into dashboards, statistics and evidence for decisions | Analysts spend their week fixing data instead of analysing it |
Quality checks deserve more attention than they usually get. The Government Data Quality Framework sets out six dimensions to measure: completeness, uniqueness, consistency, timeliness, validity and accuracy[2]. Each can become an automated test that runs on every load, so a broken feed is caught at six in the morning rather than in front of a select committee.
APIs are how a data platform avoids becoming another silo. Government's API standards recommend producing an OpenAPI document as the first output of the design process and using OAuth 2.0 to manage access[3]. The discipline matters more than the technology. An API without a written contract soon becomes one more integration nobody dares to change.
How do you prove the data can be trusted?
Trust has to be shown, not claimed. Four things make the difference:
- Lineage. You can trace any figure back to its source records and every step that changed it. Government's guidance on AI-ready datasets recommends lineage and impact analysis to trace where data came from and what a change will affect[4].
- Monitoring. Quality is measured all the time, not checked once at go-live. The same guidance describes data quality monitoring as an automated and continuous process[4].
- Definitions. Every key term has one written definition and a named owner, published where people can find it.
- Reconciliation. Record counts and totals are compared between source and destination on every run, and the results are kept.
None of this is glamorous. All of it is what an auditor, a data protection officer or a sceptical policy team will ask to see.
What does this look like on a real government programme?
When the Infected Blood Compensation Authority was set up, it had no technology estate at all. Working with CGI under a statement of work, Mayfair mobilised an SC cleared squad in nine days to build a secure, cloud-native data platform on Azure, using infrastructure as code, with a master data catalogue, role-based access controls and a Zero Trust blueprint. All of the authority's case data was brought together in one place, ending the spreadsheet sprawl, and near real-time reporting for ministers was live within three months. Read the IBCA case study.
At the Office for National Statistics the problem was different. Sensitive records from several departments had to be de-identified, quality checked and linked before analysts could touch them. For the Integrated Data Service we delivered anonymisation workflows with automated disclosure checks on every dataset as it is ingested, and a reference data hub that gives analysts common geographic and demographic lookups through APIs. Read the ONS case study.
Where should a public body start?
Start with one decision that matters and work backwards. Pick a figure your leadership relies on, trace where it comes from, and measure it against the six quality dimensions. That single exercise usually shows exactly which pipeline to build first, and gives you a baseline to show the improvement.
Mayfair IT Consultancy's data transformation and data migration teams build and run the data pipelines behind digital government services, with SC and DV cleared engineers, from the first ingestion job to reporting a board can rely on.
Related reading
- Public Sector Data Modernisation: From Fragmented Systems to Single Source of Truth
- Government Data Governance: Ownership, Quality & Compliance
- AI Data Platforms for Government: Why Clean Data Comes First
- All insights from Mayfair IT Consultancy
Common questions
Is data engineering the same as building a data warehouse?
A warehouse is one place data can land. Data engineering covers the whole route: getting data out of source systems, checking and reshaping it, storing it with history, and serving it through reports and APIs. Many newer platforms use a data lake or lakehouse rather than a classic warehouse, but the engineering disciplines are the same.
Do we have to replace legacy systems before we can trust our data?
No. Well-built pipelines read from legacy systems without changing them, which gets trusted data out now. It also makes the eventual migration safer, because by then you understand the data properly.
How long does it take to build a first trusted data pipeline?
For a single, well-scoped source feeding one decision, weeks rather than months is often realistic. The Infected Blood Compensation Authority went from no estate to near real-time ministerial reporting within three months, though a fresh start is easier than unpicking decades of legacy.
What security clearance do data engineers need for government work?
It depends on the data. Much public sector data needs BPSS or SC cleared staff, and the most sensitive programmes need DV. Agree the level with your security team before work starts, because clearance can take longer than the build.