Why we’ve decided to decommission Gov.uk PaaS(gds.blog.gov.uk)
gds.blog.gov.uk
Why we’ve decided to decommission Gov.uk PaaS
https://gds.blog.gov.uk/2022/07/12/why-weve-decided-to-decommission-gov-uk-paas-platform-as-a-service/
162 comments
A pathetic example of the ideological myopia that plagues Europe these days. States have provided services to their inhabitants since prehistory, and the notion that this infrastructure does not need financing is as healthy as a corporation that thinks it does not need to invest to succeed. At best, this strategy of living off of investments made by previous generations is doomed to underperform compared to states that take an active role in their own development.
We don’t even have to zoom in on Europe. In the US, it’s quite common for folks to expect public services (like public transit or even the USPS) to be profitable. Which means these services don’t get the investment they really need.
If it was providing a service of as much value as was being spent on it, then why shouldn't it be profitable?
Having a profit motive drives a behavior of raising prices as high as the market will bear. That inherently excludes citizens who cannot afford to pay the “most efficient” price, who are often the ones who need the service the most (e.g. busses and other public transport). That reduces the ability for society to operate as a whole. If someone can’t afford to get to school, or to an entry-level job, how will they ever be able to move up in skills and pay? Who will the high-end electronic store sell their products to if people can’t afford them?
Municipal services are meant to serve as many citizens as possible, not to extract as much profit as possible, because it’s an overall benefit to everyone.
Municipal services are meant to serve as many citizens as possible, not to extract as much profit as possible, because it’s an overall benefit to everyone.
Public services are profitable. You just need to look at the systemic effects of having the services available instead of just dollar signs
Just looking at money is incredibly short sighted and naïve
Just looking at money is incredibly short sighted and naïve
> If it was providing a service of as much value as was being spent on it, then why shouldn't it be profitable?
The entire point of a public service is that benefits accrue to people other than the direct consumers of the service.
The entire point of a public service is that benefits accrue to people other than the direct consumers of the service.
What percentage of US taxpayer money should go to subsidize money-losing services?
An amount equal to the external utility of the services.
As much as necessary.
I don't know if this is a European wide problem or just a UK one.
Definitely applies to Germany. Our infrastructure is deeply fucked and we missed all the chances to innovate after 16 years of conservative rule, but the balance sheets look nice though.
It certainly seems worst in the UK.
I was the first user of the GOV.UK PaaS for a production service - GOV.UK Trade Tariff [0]. It's unfortunate that it has come to this, and as others have pointed out, the blog doesn't instil confidence about using other GDS services in the future.
Procurement in government is painfully slow, and that's one thing that the PaaS excelled at, all departments could get set up immediately.
I hope when making their financial assessment, they included the saving that all departments using the PaaS are making not in infrastructure but ops. Each service will increase ops spending; there isn't a comparable managed platform for UK GOV services, so each service will have to increase its devops support costs. I've used Google Cloud Run and AWS Copilot, and both are far from stable enough, in my opinion, for production use.
My only hope is that they revisit this decision, but it seems likely we'll have to move the services we manage off.
[0] https://www.ukauthority.com/articles/trade-tariff-moves-to-g...
Procurement in government is painfully slow, and that's one thing that the PaaS excelled at, all departments could get set up immediately.
I hope when making their financial assessment, they included the saving that all departments using the PaaS are making not in infrastructure but ops. Each service will increase ops spending; there isn't a comparable managed platform for UK GOV services, so each service will have to increase its devops support costs. I've used Google Cloud Run and AWS Copilot, and both are far from stable enough, in my opinion, for production use.
My only hope is that they revisit this decision, but it seems likely we'll have to move the services we manage off.
[0] https://www.ukauthority.com/articles/trade-tariff-moves-to-g...
I've used the GOV.UK PaaS since the private beta. I was a user research subject and gave early feedback. I've used it on multiple projects since.
As with Notify, the Design System, and all the other great tools that GDS produces, the PaaS is an absolute joy to use. Think Heroku, but built on top of FOSS, and with everything you could want to help you build the long tail of cookie cutter form or API-based government services.
It's a damn shame GDS killed it.
It's even worse they did so without providing a clear migration path. That sends a very bad message to other departments: GDS will kill things you depend on. They will also leave you high and dry when they do so. So don't depend on or adopt their tools. "Not invented here" has always been a problem in the UK government, this news only makes it worse.
Colleagues of mine have staked their reputation on the line to convince legacy IT to let us use the PaaS instead of their own home-grown, poorly documented, and unsatisfactory solutions. Now, we look like fools.
I love working in government. I love building accessible, robust, and user-centred services.
I don't love: writing IAM policies, learning about what a "service principal" is, naming resource groups, having to open a support ticket to change an environment variable, having my password expire in the middle of an incident when I want to read logs, or spending more than £50 a month of taxpayer money for a monolith with a database and a Redis. I had to do none of those on the PaaS.
RIP.
As with Notify, the Design System, and all the other great tools that GDS produces, the PaaS is an absolute joy to use. Think Heroku, but built on top of FOSS, and with everything you could want to help you build the long tail of cookie cutter form or API-based government services.
It's a damn shame GDS killed it.
It's even worse they did so without providing a clear migration path. That sends a very bad message to other departments: GDS will kill things you depend on. They will also leave you high and dry when they do so. So don't depend on or adopt their tools. "Not invented here" has always been a problem in the UK government, this news only makes it worse.
Colleagues of mine have staked their reputation on the line to convince legacy IT to let us use the PaaS instead of their own home-grown, poorly documented, and unsatisfactory solutions. Now, we look like fools.
I love working in government. I love building accessible, robust, and user-centred services.
I don't love: writing IAM policies, learning about what a "service principal" is, naming resource groups, having to open a support ticket to change an environment variable, having my password expire in the middle of an incident when I want to read logs, or spending more than £50 a month of taxpayer money for a monolith with a database and a Redis. I had to do none of those on the PaaS.
RIP.
Agree, I'm not sure how they can say with a straight face that people should use the other new forms services they are launching. If that is not profitable will it just be shut down in a few years too?
I am wondering if there are political force behind it, that big Cloud Provider wants the Government to shut down their own PaaS.
PaaS ran on AWS anyway, so it wouldn't be Amazon (I would guess? Unless they think they can charge more/upsell by departments procuring directly from them). Perhaps Google or MS were upset that by PaaS running atop AWS, AWS became the de facto 'official' cloud service for government?
I think it has to be intentional, but if not it’s still funny :)
> ”Over the same period departments have built better and more expert in-house cloud engineering capability, and are (broadly) clustering around a Kubernetes based architecture.”*
> ”Over the same period departments have built better and more expert in-house cloud engineering capability, and are (broadly) clustering around a Kubernetes based architecture.”*
Another Cloud foundry based platform moving on to Kubernetes. Perhaps https://github.com/cloudfoundry/cf-for-k8s can provide a simple migration path..
Have you seen Korifi? https://github.com/cloudfoundry/korifi
This is the one I expect to become the primary CF product on k8s
This is the one I expect to become the primary CF product on k8s
Ah, another one! On the other hand, if you've already containerized your app and are not using buildpacks, the switch to Kubernetes should be fairly easy. Thanks for the link, will check.
They mention a handful of services like GOV.UK Notify, GOV.UK Pay, the GOV.UK Design System, the Prototype Kit, and the new GOV.UK Forms product.
Is anyone able to clarify whether these things are collectively referred to or considered part of the Gov.uk PaaS offering (and therefore will all be decomissioned) or not? It wasn't clear to me from the article and I'm not otherwise familiar.
Is anyone able to clarify whether these things are collectively referred to or considered part of the Gov.uk PaaS offering (and therefore will all be decomissioned) or not? It wasn't clear to me from the article and I'm not otherwise familiar.
My understanding is those are services various government departments can make use of. Notify being to send notifications to people, Pay to take payments, etc. These are basically SaaS offerings from GDS to over government departments.
The GOV.UK PaaS is a hosting platform for other government departments to host their own software in the GOV.UK managed platform (like a Heroku kind of thing).
They're investing more in their SaaS services (Notify, Pay, Forms, Prototyping Kit, etc) and deprecating their PaaS offering in favour of Kubernetes, which looks like it may be offered as a managed service for government departments to use. It reads to me like more of a v2 of the PaaS but it'll now be kube based rather than whatever homegrown thing it is at the moment.
Edit: as tnwhitwell pointed out below, the PaaS being deprecated uses CloudFoundry and isn't homegrown.
The GOV.UK PaaS is a hosting platform for other government departments to host their own software in the GOV.UK managed platform (like a Heroku kind of thing).
They're investing more in their SaaS services (Notify, Pay, Forms, Prototyping Kit, etc) and deprecating their PaaS offering in favour of Kubernetes, which looks like it may be offered as a managed service for government departments to use. It reads to me like more of a v2 of the PaaS but it'll now be kube based rather than whatever homegrown thing it is at the moment.
Edit: as tnwhitwell pointed out below, the PaaS being deprecated uses CloudFoundry and isn't homegrown.
Just a small correction, it’s not a homegrown thing, but a CloudFoundry deployment with a bunch of tooling around it
Ah yes I missed that bit, thanks for the correction.
They're not, no. Think of PaaS as like Heroku, whereas Notify us more like Sendgrid (ie a SaaS) and DesignSysten is like MaterialUI.
Most services are not using PaaS, but are deployed onto AWS or Azure and integrate with other services (notably Notify) via APIs.
Most services are not using PaaS, but are deployed onto AWS or Azure and integrate with other services (notably Notify) via APIs.
Notify, Pay, Design System, Prototype Kit and Forms are unaffected by this news.
I think the PaaS does not include them; it's the service for hosting a service.[1]
[1] https://www.cloud.service.gov.uk/
[1] https://www.cloud.service.gov.uk/
Those were originally part of the GaaP (Government as a Platform) programme. The PaaS is explicitly just the Cloud Foundry-based deployment platform.
We use GovPaas for a few things. GovPaas provides the platform which we deploy our applications onto (under the hood it's Cloud Foundry).
It is incredibly cheap for our use cases (as it turns out, unsustainably so for the team who provide it!). Our interpretation of this news is that we will need to migrate away to another platform.
It is incredibly cheap for our use cases (as it turns out, unsustainably so for the team who provide it!). Our interpretation of this news is that we will need to migrate away to another platform.
No PaaS is only the hosting
Key paragraph: "In parallel, we are starting a piece of joint work with the Central Digital & Data Office, in partnership with Chief Technology Officers (CTOs) across government, to understand what a future central hosting offer could or should be. We don’t know what we’ll conclude, the options ranging from doing nothing, to creating a reusable set of configuration and management components (similar to the GOV.UK Design System, but for secure cloud hosting) all the way through to building a new PaaS v2 using different architecture."
You'd think that this is something that would have be done before announcing the decommissioning in 18 months.
Indeed. Perhaps the purpose of this message was to let all the projects know they get no advice but a permission and freedom to migrate anywhere. After a while, there may not be anyone to use a PaaS v2 anymore.
Something which won't exactly inspire confidence for any similar attempts in the future
I don't get it, are they saying that public organizations should move to using AWS/Azure/GCP?
In a way, it could be interpreted like that.
It's another way of saying "maintaining infrastructure isn't a task of a public service." Generally, the "why/why not" tends to be answered with: "A public services can never provide a similar value proposition compared to private vendors." which isn't really an answer to the question.
One problematic aspect is that public entities don't just rely on infrastructure, but also on an entire stack of in-house expertise that provides support and understands the specifics that come with the problem domain - e.g. legal compliance, accountability,... - of providing public services online.
So, barring an alternative, this would effectively mean that public services would have to source consultancy themselves on the private market to set up / maintain their applications on a private cloud on a per-project basis. Both of which are easily a factor more expensive compared to a shared platform / framework / solution where resources and expertise can be pooled together to provide affordable and secure services towards other branches of the authorities.
Oddly enough, they seem to be all to aware of that conundrum:
> In parallel, we are starting a piece of joint work with the Central Digital & Data Office, in partnership with Chief Technology Officers (CTOs) across government, to understand what a future central hosting offer could or should be. We don’t know what we’ll conclude, the options ranging from doing nothing, to creating a reusable set of configuration and management components (similar to the GOV.UK Design System, but for secure cloud hosting) all the way through to building a new PaaS v2 using different architecture.
Personally, I believe this is a decision based on cost. The cost of keeping centralized hosting infrastructure based on old technology online, or migrating towards new technology, is just too steep and can't be readily justified anymore in 2022 compared to what the private market offers. Perhaps most surprisingly: there's no clear short-term plan B for all those entities relying on this service.
If I were managing a few projects from some department, I wouldn't be happy learning that I'd now have 18 months to scramble, figure out how to move forward, find funds, write a project proposal, align all stakeholders, source consultancy, etc. without going offline for a prolonged time.
It's another way of saying "maintaining infrastructure isn't a task of a public service." Generally, the "why/why not" tends to be answered with: "A public services can never provide a similar value proposition compared to private vendors." which isn't really an answer to the question.
One problematic aspect is that public entities don't just rely on infrastructure, but also on an entire stack of in-house expertise that provides support and understands the specifics that come with the problem domain - e.g. legal compliance, accountability,... - of providing public services online.
So, barring an alternative, this would effectively mean that public services would have to source consultancy themselves on the private market to set up / maintain their applications on a private cloud on a per-project basis. Both of which are easily a factor more expensive compared to a shared platform / framework / solution where resources and expertise can be pooled together to provide affordable and secure services towards other branches of the authorities.
Oddly enough, they seem to be all to aware of that conundrum:
> In parallel, we are starting a piece of joint work with the Central Digital & Data Office, in partnership with Chief Technology Officers (CTOs) across government, to understand what a future central hosting offer could or should be. We don’t know what we’ll conclude, the options ranging from doing nothing, to creating a reusable set of configuration and management components (similar to the GOV.UK Design System, but for secure cloud hosting) all the way through to building a new PaaS v2 using different architecture.
Personally, I believe this is a decision based on cost. The cost of keeping centralized hosting infrastructure based on old technology online, or migrating towards new technology, is just too steep and can't be readily justified anymore in 2022 compared to what the private market offers. Perhaps most surprisingly: there's no clear short-term plan B for all those entities relying on this service.
If I were managing a few projects from some department, I wouldn't be happy learning that I'd now have 18 months to scramble, figure out how to move forward, find funds, write a project proposal, align all stakeholders, source consultancy, etc. without going offline for a prolonged time.
Given they charged a markup on AWS costs, I find that it's weird they did not manage to at least break even. Other departments switching to running on their own cloud account rather than use the PaaS could be the main reason, and that's partly due to the PaaS/Cloud foundry stagnating.
No, not at all. They released services and software to make integrations easier on these platforms but the toolset is becoming stale and more effort goes into k8s (on several different clouds, depending on project)
Well they sort of do already. There are armies of consultancies queuing up to provide packaged solutions far above the cloud infra level solutions abstractions that GDS offer. Those guys use commercial clouds.
I wish this were spread to other government IT things.
The NHS uses a patchwork of different systems. Why should every hospital have to sort out its own IT needs. Let the NHS do NHS things and have an IT branch providing standardised software for everyone.
The NHS uses a patchwork of different systems. Why should every hospital have to sort out its own IT needs. Let the NHS do NHS things and have an IT branch providing standardised software for everyone.
Large government IT projects end up being done by companies expert in extracting as much cash from the government as possible. The end result is the product doesn't work properly and is massively over budget. Then the government is forced to hand over even more money to redesign bits to work better.
The government has been bitten by this many times, so now shys away from big IT systems. Instead every department has to build its own, usually with lots of Excel sheets and macros involved.
The government has been bitten by this many times, so now shys away from big IT systems. Instead every department has to build its own, usually with lots of Excel sheets and macros involved.
The move at the moment is for central orgs to specify standards and protocols so it doesn't matter who implements them and the delegated authorities can buy from whoever. That goes some way to preventing both one-big-vendor setups and unstandardised fragmentation, and it was blindingly obvious to anyone who's ever implemented more than a bash script that it was the correct approach even 20 years ago. Only it's taken a generation of civil servants ageing out at the top end to make it happen.
This is largely what Connecting for Health attempted, and the only good things that came out of it were intangible to most - N3 and the UK SNOMED extensions.
Unfortunately this was largely down to incompetent consultancies, and a focus on providing what management wanted rather than what clinicians wanted, and it has absolutely poisoned the well with regards to information technology in the NHS.
Unfortunately this was largely down to incompetent consultancies, and a focus on providing what management wanted rather than what clinicians wanted, and it has absolutely poisoned the well with regards to information technology in the NHS.
i was unable to update my driving licence for the best part of a year because of some (still unknown) internal issue.
i hope they have better luck bringing back up a degraded/read-only etcd service after a power cut.
unrelated - i guess i have some real contempt for the current deployment stack meta :/
i hope they have better luck bringing back up a degraded/read-only etcd service after a power cut.
unrelated - i guess i have some real contempt for the current deployment stack meta :/
DVLA systems seem to be an absolute mess, and many public-facing systems are still using the DirectGov styling (which was replaced by GOV.UK in 2012). A lot of their services are completely unavailable overnight from 7pm-7am. So if you sell a vehicle on the last day of the month and go to register the sale after 7pm, bad luck - you're getting charged another month of vehicle tax.
> We also need to operate with a growth mindset in GDS
Is it just me or is it weird to see "startup speak" from gov agencies?
Is it just me or is it weird to see "startup speak" from gov agencies?
That's not startup speak it's "business consultancy" speak and the UK goverment loves to hire business consultants to tell the uk civil service how to work.
Also though... the actual content, why? Why does GDS need to operate with a "growth mindset"?
They qualify their usage of business-speak:
> Our growth strategy is obviously not about making profit, but we are a small team at the centre of government and we need to make sure our people and our money are focused on services that have the biggest reach and the most impact.
It's not quite Fry and Laurie yet. "Oh, you have shares in the Royal Family?" https://www.youtube.com/watch?v=vLfghLQE3F4
> Our growth strategy is obviously not about making profit, but we are a small team at the centre of government and we need to make sure our people and our money are focused on services that have the biggest reach and the most impact.
It's not quite Fry and Laurie yet. "Oh, you have shares in the Royal Family?" https://www.youtube.com/watch?v=vLfghLQE3F4
[deleted]
Cynically it can be seen as part of the war on introversion.
Consultancies rebadged introverts as having a "fixed mindset" and extroverts as having a "growth mindset" and then labelled having a fixed mindset as a terrible thing.
But that criticism is demonstration of a "fixed mindset" and therefore should be discarded and should be ignored.
That's the cynical take. The less cynical take is that having an open mind helps businesses and services adapt to their needs.
Consultancies rebadged introverts as having a "fixed mindset" and extroverts as having a "growth mindset" and then labelled having a fixed mindset as a terrible thing.
But that criticism is demonstration of a "fixed mindset" and therefore should be discarded and should be ignored.
That's the cynical take. The less cynical take is that having an open mind helps businesses and services adapt to their needs.
It's not clear to me how "growth mindset" equates to "having an open mind".
It is however desirable to see compared to what came before.
[deleted]
mordae(7)
>The big cloud providers (AWS, Azure, GCP and others) have upped their game
So now UK citizens are going to be forced to use services from a country known for largely non-working legal system (where being provably innocent is not enough to call off execution, see Scalia; where a monetary fine is an adequate sentence for premeditated murder, see OG Simpson, not to mention de facto legal immunity for cops committing crimes on duty) and for droning random civilians with impunity (see Hague Invasion Act).
So now UK citizens are going to be forced to use services from a country known for largely non-working legal system (where being provably innocent is not enough to call off execution, see Scalia; where a monetary fine is an adequate sentence for premeditated murder, see OG Simpson, not to mention de facto legal immunity for cops committing crimes on duty) and for droning random civilians with impunity (see Hague Invasion Act).
That's a feature. The UK can ask the US to request data to AWS about UK citizens without breaking any law about spying on UK citizens. Now they can filter all citizens that ticked "muslim" or "arab" in one of the many gov forms, cross reference with fb/twitter/insta for signs of extremism and flag them as suspicious individuals.
Large parts of GDS already run on AWS, and I think that may even have been the case for PaaS itself.
Yes, GOV.UK PaaS runs atop AWS.
> OG Simpson
It's OJ, but thanks for giving me the perfect stage name for my 90s-pop-culture-themed hip-hop act.
It's OJ, but thanks for giving me the perfect stage name for my 90s-pop-culture-themed hip-hop act.
[0] https://www.gov.uk/service-standard-reports/gov-dot-uk-platf...