Ask HN: Cause of UK e-gates outage?
60 comments
Re: e-gates shambles
Fujitsu's contract runs from 4 September 2021 to 2024 https://www.theregister.com/2021/09/09/fujitsu_border_crossi... https://www.theregister.com/2022/01/14/uk_border_upgrade_con...
Re: Horizon disgrace
Fujitsu has also, so far, escaped financial penalties, whereas the government has been forced to set aside £1bn to cover the costs of compensating victims of the scandal. Meanwhile, Fujitsu is continuing to win significant IT projects with the UK government. https://www.computerweekly.com/news/252526102/Fujitsu-to-fin...
UK Gov & Fujitsu smells bad
Fujitsu's contract runs from 4 September 2021 to 2024 https://www.theregister.com/2021/09/09/fujitsu_border_crossi... https://www.theregister.com/2022/01/14/uk_border_upgrade_con...
Re: Horizon disgrace
Fujitsu has also, so far, escaped financial penalties, whereas the government has been forced to set aside £1bn to cover the costs of compensating victims of the scandal. Meanwhile, Fujitsu is continuing to win significant IT projects with the UK government. https://www.computerweekly.com/news/252526102/Fujitsu-to-fin...
UK Gov & Fujitsu smells bad
They honestly should be blacklisted after the Horizon scandal. Allowing them to get away with it sets a precedent that could encourage other companies to behave just as irresponsibly.
As it stands, is yet another example of privatising profits while socialising the costs, just in this case, the costs weren't (entirely) financial.
As it stands, is yet another example of privatising profits while socialising the costs, just in this case, the costs weren't (entirely) financial.
A lot of UK government IT is like this and has been forever.
There was a short period under the coalition where GDS was set up and they created a consistent framework for people to work with and would open a portion of the market up to smaller players.
Now it's mostly large bodyshop middlemen like the prime minister's wife's company taking a 70% cut for the work of a cheap offshore developer.
There was a short period under the coalition where GDS was set up and they created a consistent framework for people to work with and would open a portion of the market up to smaller players.
Now it's mostly large bodyshop middlemen like the prime minister's wife's company taking a 70% cut for the work of a cheap offshore developer.
a lot of the problem is the contract procurement rules, which are written so only large companies are capable of bidding
a legacy of EU membership, but I doubt the government has any will to change it
a legacy of EU membership, but I doubt the government has any will to change it
During COVID the government tried skipping those procurement rules in the interests of expediency. The result was a huge corruption scandal.
> During COVID the government tried skipping those procurement rules in the interests of expediency. The result was a huge corruption scandal.
I don't think that this is the guaranteed outcome for all governments and situations.
In Latvia, the COVID-19 contract tracing application "Apturi Covid" wasn't developed with a bunch of buraucracy beforehand, but as a voluntary community effort between both different orgs and people. As a consequence, it was developed reasonably quickly, actually worked and was helpful (even if the user numbers weren't as great as expected, but that's besides the point): https://lv-m-wikipedia-org.translate.goog/wiki/Apturi_Covid?...
I know this because I was a part of the developer team (the website in particular: https://apturicovid.lv/#en before handing it over to govt. once my participation was concluded) and it was actually encouraging to see how well professionals with a common goal can work together, as opposed to some of the less successful processes I've seen. In particular, our e-health platform was once of those serious government projects with lots of bureaucracy, yet still didn't work after many millions in investments: https://www-lsm-lv.translate.goog/raksts/zinas/zinu-analize/...
(I've mentioned this previously, but the point still stands and the contrast is very apparent)
I don't see why a slightly more streamlined, iterative and goal oriented process couldn't work for commercial software projects, even in the public sector. Of course, if the clients don't know what they want and change their requirements whenever new people come into the office (new management), no particular process is going to save you. Nor will you be successful when people quite frankly don't care about shipping software that works and instead of answering your questions just throw a PDF with 200 pages at you, that doesn't provide the actual answer either.
I don't think that this is the guaranteed outcome for all governments and situations.
In Latvia, the COVID-19 contract tracing application "Apturi Covid" wasn't developed with a bunch of buraucracy beforehand, but as a voluntary community effort between both different orgs and people. As a consequence, it was developed reasonably quickly, actually worked and was helpful (even if the user numbers weren't as great as expected, but that's besides the point): https://lv-m-wikipedia-org.translate.goog/wiki/Apturi_Covid?...
I know this because I was a part of the developer team (the website in particular: https://apturicovid.lv/#en before handing it over to govt. once my participation was concluded) and it was actually encouraging to see how well professionals with a common goal can work together, as opposed to some of the less successful processes I've seen. In particular, our e-health platform was once of those serious government projects with lots of bureaucracy, yet still didn't work after many millions in investments: https://www-lsm-lv.translate.goog/raksts/zinas/zinu-analize/...
(I've mentioned this previously, but the point still stands and the contrast is very apparent)
I don't see why a slightly more streamlined, iterative and goal oriented process couldn't work for commercial software projects, even in the public sector. Of course, if the clients don't know what they want and change their requirements whenever new people come into the office (new management), no particular process is going to save you. Nor will you be successful when people quite frankly don't care about shipping software that works and instead of answering your questions just throw a PDF with 200 pages at you, that doesn't provide the actual answer either.
There have also been huge corruption scandals before and after COVID. Grift will find a way.
clearly there's a middle ground
Except during the coalition things were far better with GDS, while we were in the EU, after the new-labour ppp stuff and before the tory-grift stuff.
They don't call them Failjitsu for no reason.
It's not just UK. They've been terrible everywhere. Maybe just the cheapest bid.
It's not just UK. They've been terrible everywhere. Maybe just the cheapest bid.
Underpaid third-world slave working for whatever shitty consultancy got the contract pushed the wrong button.
It's "too sensitive to discuss" because it would be inconvenient to admit that the people who have privileged access to this system aren't paid enough to give a shit and are very vulnerable to bribery.
It's "too sensitive to discuss" because it would be inconvenient to admit that the people who have privileged access to this system aren't paid enough to give a shit and are very vulnerable to bribery.
> whatever consultancy pushed the wrong button.
So, the usual then?
https://www.theregister.com/2017/06/02/british_airways_data_...
So, the usual then?
https://www.theregister.com/2017/06/02/british_airways_data_...
higo(1)
Too sensitive to discuss can also mean too embarrassing to discuss
They watched "Yes, Minister" and took it as a guide:
https://www.youtube.com/watch?v=6Y4PEqvk0Jg
https://www.youtube.com/watch?v=6Y4PEqvk0Jg
No idea but I’m gonna go for expired TLS cert, and Finance won’t let you have the corporate card to renew it until Monday when the approver is back from holiday.
> Until Monday when the approver is back from holiday.
Until Tuesday. Monday 29th May is a public holiday in the UK.
Until Tuesday. Monday 29th May is a public holiday in the UK.
[deleted]
No insight here, but I've always wondered if the e-gates are doing anything "clever" with the camera, or if they're just connecting you to some call-centre in Swindon where your face and passport details pop up in front of someone with a lot less training than traditional border security.
It might be doing facial recognition, but it feels too reliable for the level of facial recognition I expect a consultancy could pull off in a government contract.
It might be doing facial recognition, but it feels too reliable for the level of facial recognition I expect a consultancy could pull off in a government contract.
I don't have a full answer but what I observed is that there is always someone that is nearby and "in control".
The base I think works like this:
- you scan the passport
- you look at the camera
- your passport details & photo are sent to the control room with your live feed
- a guy looks at everything those details plus your immigration history and other info
- same guy decides to let you in or send you to a desk
This was particularly obvious in the old Gibraltar border where you could easily be the only person going through and so the agent had to do the routine just to let you pass.
The base I think works like this:
- you scan the passport
- you look at the camera
- your passport details & photo are sent to the control room with your live feed
- a guy looks at everything those details plus your immigration history and other info
- same guy decides to let you in or send you to a desk
This was particularly obvious in the old Gibraltar border where you could easily be the only person going through and so the agent had to do the routine just to let you pass.
Yeah this is pretty much my assumption. I would imagine that the person doing the judging may not be on site as it's probably cheaper and easier to have a bigger workforce in one office, flex the people in shifts, load balance across all ports with the gates, etc.
I do wonder how much is automated. For example, is the gate fully automated in calculating a risk score and then referring you to a border agent if above a threshold? Is a person looking at the details in realtime to decide that, or are they just doing facial recognition, or are they only involved when the gates fail.
I do wonder how much is automated. For example, is the gate fully automated in calculating a risk score and then referring you to a border agent if above a threshold? Is a person looking at the details in realtime to decide that, or are they just doing facial recognition, or are they only involved when the gates fail.
I don't know anything about this particular case, but about these systems in general.
They can work without any operator or network connection. They verify that you have a valid passport and that the taken image (face) matches the one stored in your passport.
But for modern passports they use Extended Access Control which requires up to date terminal certificates to access the data (you have to update them in the range of days) und you can give these systems revocation lists and lists of unwanted persons. If any of this is not updated, they stop working.
> But for modern passports they use Extended Access Control which requires up to date terminal certificates to access the data (you have to update them in the range of days
Passports don't know the current time and thus can't tell whether the presented certificate is within its validity range (as in a malicious attacker could feed an expired certificate as well as a fake "current time" value to make it appear valid), so why are those certificates short-lived?
Passports don't know the current time and thus can't tell whether the presented certificate is within its validity range (as in a malicious attacker could feed an expired certificate as well as a fake "current time" value to make it appear valid), so why are those certificates short-lived?
Whenever you present a certificate to the passport, its current time is updated by the "valid from" value if it's newer then the current time.
It's not perfect but if you started you trip in another country with such a system and where a more recent certificate was used, your passport will deny access.
It's not perfect but if you started you trip in another country with such a system and where a more recent certificate was used, your passport will deny access.
Those stupid gates have never worked for me, and I've always suspected that they are actually just connected to a call-centre type operation. Given how fast my phone can recognise my face (in 3d no less), the e-gates are a total joke.
That's so weird, I use them regularly and have never had a problem in the UK or other countries with them.
I wonder if you're failing due to not recognising your face, or (no offence intended) that you have too high a risk profile in some way due to your passport info, and they decide they want to see you in person.
I wonder if you're failing due to not recognising your face, or (no offence intended) that you have too high a risk profile in some way due to your passport info, and they decide they want to see you in person.
I have always assumed face recognition because my passport picture has no glasses and the gates never work when I wear my glasses. A human would recognise me, I think.
Interesting. I'm the opposite, my passport photo doesn't have glasses, but I've never taken them off for the gates and I don't think they tell you to, and I've never had a gate fail.
They do tell you to take them off. But they might be targeting shades more than regular glasses.
Oh! How weird. Maybe I'm getting the UK gates mixed up with Australia where I'm often travelling to. I do remember looking for a sign and being surprised that I didn't need to take them off recently, but it might have been outbound.
Nothing specific, but I've seen some big systems from the inside, and I know the kind of thing that leads to failures:
* Back in the 60s they got a big IBM computer to do some stuff. Then later on they needed to do other stuff. The old computer was too expensive and difficult to replace, so they got a new VAX or something to do the new stuff and talk to the old mainframe. Then some PCs got added to do more stuff, and so on. Today the back end consists of many different systems of different ages all talking to each other using different protocols that were designed against different requirements. Newer systems are forever being patched and updated to cope with new requirements, while the code for old requirements lurks waiting to be accidentally reactivated. Each of these systems has its own specialists for care and feeding, but nobody fully understands the whole thing. When something goes down there are not many people who can diagnose the fault and get it back up.
* Government contracts have lots of rules around them to ensure value for money and prevent corruption (see the UK COVID PPE fiasco for what happens when you try to cut these rules out). But the size and complexity make even bidding for a big contract very expensive and complicated, so it tends to be the preserve of a few big companies who chose to specialise in it. Their core competence is winning these contracts, not delivering on them later.
* These rules mean that everything has to be specified in detail up front, so that everybody knows what is supposed to happen. But this makes the whole thing horribly inflexible. As new requirements emerge from the woodwork there is a continuous process of renegotiation.
* The UK civil service is based around the "cult of the gifted amateur". Senior managers are rotated around departments every few years. So the person who kicks off a project is rarely the person who sees it through. Everybody gets to blame someone else for failure.
* When one of these big contractors fails to deliver, the Government has to chose between suing to get their money back (or some of it) in a few years, or getting the at least part of the system they actually need at a higher price. The government doesn't need the money, it needs the system. So the contractor gets to carry on regardless of failure.
* Humans are very bad at managing small risks with large consequences. Many big disaster stories have at their heart someone who decided that the risk was too small to be bothered with.
* Back in the 60s they got a big IBM computer to do some stuff. Then later on they needed to do other stuff. The old computer was too expensive and difficult to replace, so they got a new VAX or something to do the new stuff and talk to the old mainframe. Then some PCs got added to do more stuff, and so on. Today the back end consists of many different systems of different ages all talking to each other using different protocols that were designed against different requirements. Newer systems are forever being patched and updated to cope with new requirements, while the code for old requirements lurks waiting to be accidentally reactivated. Each of these systems has its own specialists for care and feeding, but nobody fully understands the whole thing. When something goes down there are not many people who can diagnose the fault and get it back up.
* Government contracts have lots of rules around them to ensure value for money and prevent corruption (see the UK COVID PPE fiasco for what happens when you try to cut these rules out). But the size and complexity make even bidding for a big contract very expensive and complicated, so it tends to be the preserve of a few big companies who chose to specialise in it. Their core competence is winning these contracts, not delivering on them later.
* These rules mean that everything has to be specified in detail up front, so that everybody knows what is supposed to happen. But this makes the whole thing horribly inflexible. As new requirements emerge from the woodwork there is a continuous process of renegotiation.
* The UK civil service is based around the "cult of the gifted amateur". Senior managers are rotated around departments every few years. So the person who kicks off a project is rarely the person who sees it through. Everybody gets to blame someone else for failure.
* When one of these big contractors fails to deliver, the Government has to chose between suing to get their money back (or some of it) in a few years, or getting the at least part of the system they actually need at a higher price. The government doesn't need the money, it needs the system. So the contractor gets to carry on regardless of failure.
* Humans are very bad at managing small risks with large consequences. Many big disaster stories have at their heart someone who decided that the risk was too small to be bothered with.
I’m not sure if all the gates are by the same vendor, but at least one of them is vision-box[1]. My suspicion is there’s a backend system on the Home Office’s side that went down. The ICIBI did an assessment of the epassport gate system recently, detailed here [2].
[2] https://assets.publishing.service.gov.uk/government/uploads/...
[1] https://www.vision-box.com/
[2] https://assets.publishing.service.gov.uk/government/uploads/...
[1] https://www.vision-box.com/
Some pod in CrashLoopBackOff somewhere.
Nah, just kidding, but the fact that it's country-wide indicates it's probably a server issue.
Nah, just kidding, but the fact that it's country-wide indicates it's probably a server issue.
There was an issue with France’s system too at Dover. It wouldn’t be tin foil hat territory to suggest a coordinated cyber attack.
I wonder who has snuck through the chaos. UK border control regularly fails when the systems up, god knows what about when they’re down
I wonder who has snuck through the chaos. UK border control regularly fails when the systems up, god knows what about when they’re down
Could just be that both systems need to communicate to each other (when it comes to adjacent-country border control it's not a bad idea to share data about potential threats or intruders) and one system didn't properly handle failures to communicate with the other system.
Or they both used the same vendor and had the same issue.
[deleted]
I'm not sure why this should affect holidaymakers, if I remember correctly passports are checked on the way in, but not when going out of the UK correct?
Holidaymakers tend to return from holiday as much as they go.
Outbound flights can be delayed too if people can’t disembark from planes because the terminal is over capacity in the arrivals side.
Outbound flights can be delayed too if people can’t disembark from planes because the terminal is over capacity in the arrivals side.
Yes but the bank holiday is the Monday, they would be returning on Monday, maybe Tuesday but not today
Many holiday changeover days at Saturday
Holidaymakers need to return home eventually.
“Too sensitive to discuss” very likely mean cyber attack
Why would it be though? What is sensitive about signalling in high-level terms cyber attack is suspected? If the worry is that you might be revealing your sources to the perpetrator, then wouldn't mentioning "too sensitive to discuss" already be as revealing to the perpetrator.
These things have never worked for me, the camera goes up and down like mad for a minute then I have to go and see a person. What a waste of time
The ones at Paris last year on Eurostar were so slow they were takeing at least 45, but with retries average of about 60 seconds to process someone. I think there were 5.
There was only 1 train an hour leaving, but the trains had 900 seats. You can see the problem.
With airports, I used to come in to Heathrow 50 times a year. There was a great system called Iris, you literally walked up and walked through, barely needed to stop. It never failed on me.
They got rid of it, partly because it wouldn't scale (as it was literally just your Iris pattern, not a pattern + passport), so they only offered it to frequent users.
There was only 1 train an hour leaving, but the trains had 900 seats. You can see the problem.
With airports, I used to come in to Heathrow 50 times a year. There was a great system called Iris, you literally walked up and walked through, barely needed to stop. It never failed on me.
They got rid of it, partly because it wouldn't scale (as it was literally just your Iris pattern, not a pattern + passport), so they only offered it to frequent users.
Last time I used them was at Gare du Nord, most of the people on our train had to go see the one guy to manually review the passport.
One thing that would massively cut passport queues and save money would be for the UK to join Schengen.
They probably found a stupid bug. Like it doesn't check expiry dates on passports.
Maybe they bought an upgrade from someone who donated to the CONservatives?
I've seen them running their stuff on Heroku, maybe nobody told them the free tier was going
Anyone with inside information they'd like to share?