The Worst U.S. Maritime Disaster in Decades(vanityfair.com)
vanityfair.com
The Worst U.S. Maritime Disaster in Decades
https://www.vanityfair.com/news/2018/04/inside-el-faro-the-worst-us-maritime-disaster-in-decades
174 comments
The bit you skipped:
> He went down to his stateroom after his conversation with Schultz, and when he returned to the bridge he said, “All right, I just sent up the latest weather. Let us clear everything off the chart table with the exception of the charts.” Schultz opened the B.V.S. program. As it happened, according to the N.T.S.B. report, because of a software glitch, the map that appeared was the very same map that had come in with the previous download, six hours earlier. The raw data on which it was based was at least 12 hours old.
It sounds like it wouldn't actually have helped if the captain had the per-hour update, because this undetailed software glitch meant the device wasn't showing the updated maps anyway.
The apparent moral of that part of the story is that the B.V.S. charts are past useless and into actively harmful territory, since they may not just be old but instead actively incorrect.
> He went down to his stateroom after his conversation with Schultz, and when he returned to the bridge he said, “All right, I just sent up the latest weather. Let us clear everything off the chart table with the exception of the charts.” Schultz opened the B.V.S. program. As it happened, according to the N.T.S.B. report, because of a software glitch, the map that appeared was the very same map that had come in with the previous download, six hours earlier. The raw data on which it was based was at least 12 hours old.
It sounds like it wouldn't actually have helped if the captain had the per-hour update, because this undetailed software glitch meant the device wasn't showing the updated maps anyway.
The apparent moral of that part of the story is that the B.V.S. charts are past useless and into actively harmful territory, since they may not just be old but instead actively incorrect.
Sounds like the mapping software should put a big red timestamp in the middle of the map if the current data is over 1 hour old.
It's unfortunate when programmers overlook something this big.
It's unfortunate when programmers overlook something this big.
Yes, I saw the same thing. Even when I've had people reviewing logs, I make the date prominent and train them to check it first. BVS's data is far more critical.
(I learned from a burnt hand: We engaged a non-technical but reliable user to check the daily backup log for errors and report any to us. One day we needed the backup and discovered it hadn't run at all for many weeks. Ouch. I asked the user; she said she indeed checked the logs daily and they were fine. She was right: She was seeing the log from the last backup, unchanged every day. My fault entirely, not hers: We should have anticipated the date problem, and we should have utilized someone technically literate enough to understand what they were reading - in this case, someone who would recognize an 'obvious' problem such as the numbers of files and bytes not changing. And we should have tested our backups more often, but that old lesson almost isn't worth mentioning.)
(I learned from a burnt hand: We engaged a non-technical but reliable user to check the daily backup log for errors and report any to us. One day we needed the backup and discovered it hadn't run at all for many weeks. Ouch. I asked the user; she said she indeed checked the logs daily and they were fine. She was right: She was seeing the log from the last backup, unchanged every day. My fault entirely, not hers: We should have anticipated the date problem, and we should have utilized someone technically literate enough to understand what they were reading - in this case, someone who would recognize an 'obvious' problem such as the numbers of files and bytes not changing. And we should have tested our backups more often, but that old lesson almost isn't worth mentioning.)
Not that it will comfort you much but you are hardly alone in this. Bad backups, logging on the same servers as where access takes place and single points of failure in personnel are some of the most frequently occurring things I come across in my 'day job'.
Thanks, and I know it well; that was an early lesson. Backups in particular are an amazing cesspool of problems for something so conceptually simple.
That's the thing that always bugs me, the vast majority of the items that I end up with on the todo list after a review would cost $0 or very little to get right.
Super frustrating. And you can't even rely on things staying fixed either, you have to review periodically or it will be back to square #1 within the year.
Super frustrating. And you can't even rely on things staying fixed either, you have to review periodically or it will be back to square #1 within the year.
> the vast majority of the items that I end up with on the todo list after a review would cost $0 or very little to get right
Agreed, and I drive people crazy with my focus on those things. Thorough design and implementation (including testing) up front cost far less than correcting problems later, and they don't add the enormous cost of downtime and other failures.
But ... I've found that human beings, even serious professionals, have a capacity limit for details, and it's not very high; and if it's for an over-the-horizon risk, attention is very limited. That is my biggest constraint, editing down the details, organizing them, automating them, and making trade-offs to reduce them to a point where others don't throw up their hands. Also, it's hard to get the budget for that up front investment in what looks to others like obsessiveness (it's not; it's carefully considered ROI).
So when you show up for your review (I don't know exactly what you do, but I have an impression), 1,000 details might have been addressed but 50 overlooked. or 1,050 details might have been implemented but there was no capacity for the next 100 - resources ran out, something else came up, etc.
So I can see it both ways.
Agreed, and I drive people crazy with my focus on those things. Thorough design and implementation (including testing) up front cost far less than correcting problems later, and they don't add the enormous cost of downtime and other failures.
But ... I've found that human beings, even serious professionals, have a capacity limit for details, and it's not very high; and if it's for an over-the-horizon risk, attention is very limited. That is my biggest constraint, editing down the details, organizing them, automating them, and making trade-offs to reduce them to a point where others don't throw up their hands. Also, it's hard to get the budget for that up front investment in what looks to others like obsessiveness (it's not; it's carefully considered ROI).
So when you show up for your review (I don't know exactly what you do, but I have an impression), 1,000 details might have been addressed but 50 overlooked. or 1,050 details might have been implemented but there was no capacity for the next 100 - resources ran out, something else came up, etc.
So I can see it both ways.
Good stuff, thank you, I can see there might be some way to get a process in place to avoid these relapses.
Been there, done that.
I recently helped a company big enough that everyone here would recognize the name fix a lot of items around monitoring and logging after finding they were running an important production system in such a manner as to be essentially flying blind. Yeah, those fatal errors in the logs just might be important...
I recently helped a company big enough that everyone here would recognize the name fix a lot of items around monitoring and logging after finding they were running an important production system in such a manner as to be essentially flying blind. Yeah, those fatal errors in the logs just might be important...
Also note that B.V.S did appear to have a 'Hourly Update' feature that did not appear to be subscribed to.
It's also important to realize that the captain was confident with the plan- except the storm in question was abnormal. That's an important fact to consider, as it throws the mariners heuristics off deciding how much more attention to pay.
It's also important to realize that the captain was confident with the plan- except the storm in question was abnormal. That's an important fact to consider, as it throws the mariners heuristics off deciding how much more attention to pay.
> Also note that B.V.S did appear to have a 'Hourly Update' feature that did not appear to be subscribed to.
From the article, "The Coast Guard report also noted that “El Faro crew did not take advantage of B.V.S.’s tropical update feature,” which would have provided hourly updates."
So hourly updates apparently were available even if not subscribed to.
From the article, "The Coast Guard report also noted that “El Faro crew did not take advantage of B.V.S.’s tropical update feature,” which would have provided hourly updates."
So hourly updates apparently were available even if not subscribed to.
There’s an interesting ethics question here: If your ship’s location places you near a hurricane, should they just quietly upgrade you for free tenporarily so that you don’t sink?
They should at least place a large warning stating that the data is not safe to use for navigational purposes.
I'm sort of surprised that data from a source like that is used. Have you ever seen the weather data that airline captains use? It looks like this:
KORD 050251Z 26006KT 10SM SCT110 M02/M09 A3019 RMK AO2 SLP232 T10171089 53009
That's the current weather conditions for Chicago O'Hare.
Looks like gibberish to you and me but airline captains know what it means and they know how old the observation is. Captain has ultimate authority on the ship, but also ultimate responsibility. It was his job to know what that weather report meant. And surprising to me if he didn't, since by other accounts he was experienced, organized, and safety-conscious.
KORD 050251Z 26006KT 10SM SCT110 M02/M09 A3019 RMK AO2 SLP232 T10171089 53009
That's the current weather conditions for Chicago O'Hare.
Looks like gibberish to you and me but airline captains know what it means and they know how old the observation is. Captain has ultimate authority on the ship, but also ultimate responsibility. It was his job to know what that weather report meant. And surprising to me if he didn't, since by other accounts he was experienced, organized, and safety-conscious.
You can't get licensed as a pilot without being able to read METARS. They will not let you fly. Find me a comparable licensure requirement for marine captains.
It's been many years, so I can't refer you to a specific paragraph in 46 CFR (https://www.law.cornell.edu/cfr/text/46/part-310), but during my training as a Merchant ship's officer I was required to (a) take a Meteorology 101 class, (b) read and understand data from the weather FAX at sea and (c) when on watch, make weather observations and report them to NOAA hourly.
I think that's close enough.
I think that's close enough.
https://en.wikipedia.org/wiki/Master_mariner#United_States
or from gCaptain, which is linked elsewhere in the thread as a source of info on this particular incident:
http://gcaptain.com/mariner/
it will take years of sea time and weeks of classes to even get your Able Seaman (AB) or 100 ton master license. If you want 500/1600 ton mate or masters then even more years of sea time and months of classes. All of these requirements are outlined by the Coast Guard, but they change the rules all the time. You are never really done; every five years, when you come up for your license renewal, you will have to take some refresher courses. Sadly, none of this may be completed while you are at work, but rather while you are on your off time. It is expensive to take these classes and while sometimes your company will pay for the classes, room, and board, don’t count on it.
(edit:format)
or from gCaptain, which is linked elsewhere in the thread as a source of info on this particular incident:
http://gcaptain.com/mariner/
it will take years of sea time and weeks of classes to even get your Able Seaman (AB) or 100 ton master license. If you want 500/1600 ton mate or masters then even more years of sea time and months of classes. All of these requirements are outlined by the Coast Guard, but they change the rules all the time. You are never really done; every five years, when you come up for your license renewal, you will have to take some refresher courses. Sadly, none of this may be completed while you are at work, but rather while you are on your off time. It is expensive to take these classes and while sometimes your company will pay for the classes, room, and board, don’t count on it.
(edit:format)
Funnily enough I actually look at the KORD weather reports all the time and took the time to learn how to read pilot reports.
I think the difference is that captains are dealing with currents in the ocean and in the air. For the most part pilots are flying above things that will make the wind change in extremely short distances.
I think the difference is that captains are dealing with currents in the ocean and in the air. For the most part pilots are flying above things that will make the wind change in extremely short distances.
Interesting! Did some searching and found this[1] which outlines the parts of a reading.
[1]: https://www.wunderground.com/metarFAQ.asp
[1]: https://www.wunderground.com/metarFAQ.asp
Didn't Tesla do that for cars near evacuation zones?
The crew did not take advantage, or the company did not buy the subscription?
The NTSB report says that it was the former, they were available but the captain had not configured the software settings to receive the hourly reports.
(Obviously, and hindsight is 20-20, hourly should've been the default).
(Obviously, and hindsight is 20-20, hourly should've been the default).
[deleted]
Is that really a UI issue or misrepresented data?
If I see a timestamp on rapidly changing data I sure as hell would like to see if there's a delayed effective stamp (most stock market tickers indicate the "effective date" of their data or say clearly there's a XX min delay).
If I see a timestamp on rapidly changing data I sure as hell would like to see if there's a delayed effective stamp (most stock market tickers indicate the "effective date" of their data or say clearly there's a XX min delay).
> Is that really a UI issue?
More of a UX issue. The worst possible kind.
More of a UX issue. The worst possible kind.
Reminded me of this aircraft accident where the time delay of radar satellite datalink weather information was critical. The time delay there is minutes not hours.
https://www.youtube.com/watch?v=83uvKWJS2os
https://www.youtube.com/watch?v=83uvKWJS2os
An aviation accident case study pinpointed a similar issue with in-cockpit weather displays. In this case the delay was less than 10 minutes, but it was enough to be fatal: https://youtu.be/83uvKWJS2os
Bridge transcript, addenda and errata:
Voyage Data Recorder
https://dms.ntsb.gov/pubdms/search/document.cfm?docID=447547...
Errata to Voyage Data Recorder
https://dms.ntsb.gov/pubdms/search/document.cfm?docID=455021...
Addendum to Voyage Data Recorder
https://dms.ntsb.gov/pubdms/search/document.cfm?docID=455022...
Attachment 1 to Addendum to Voyage Data Recorder
https://dms.ntsb.gov/pubdms/search/document.cfm?docID=455023...
Voyage Data Recorder
https://dms.ntsb.gov/pubdms/search/document.cfm?docID=447547...
Errata to Voyage Data Recorder
https://dms.ntsb.gov/pubdms/search/document.cfm?docID=455021...
Addendum to Voyage Data Recorder
https://dms.ntsb.gov/pubdms/search/document.cfm?docID=455022...
Attachment 1 to Addendum to Voyage Data Recorder
https://dms.ntsb.gov/pubdms/search/document.cfm?docID=455023...
Is the audio available from the deck?
I didn't see it anywhere.
Oh man. That transcript was the single worst thing I ever read in my whole life.
Oh man. That transcript was the single worst thing I ever read in my whole life.
> She called Davidson on the house phone and told him that the hurricane was now a Category 3. He knew that already. She proposed the escape route to the south and a smooth sail on to San Juan. He rejected her suggestion.
I wonder if in the maritime world there is the equivalent of CRM (Cockpit Resource Management), as in aviation.
Too many lives were lost due to 1st officer never taking over command when the error in the commander's thinking had already become apparent or the entire aircraft was being flown, effectively, by a single infallible captain and not a team.
Lessons were learned decades ago and at least in a 1st world outfit I would like to believe that no 1st officer would hesitate to utter the words "I have control" when in such stress.
I wonder if in the maritime world there is the equivalent of CRM (Cockpit Resource Management), as in aviation.
Too many lives were lost due to 1st officer never taking over command when the error in the commander's thinking had already become apparent or the entire aircraft was being flown, effectively, by a single infallible captain and not a team.
Lessons were learned decades ago and at least in a 1st world outfit I would like to believe that no 1st officer would hesitate to utter the words "I have control" when in such stress.
There is: bridge resource management. It actually grew out of CRM. Section 2.7 of the NTSB report (linked elsewhere in this thread) gives an overview of it and discusses how it played a role in this incident. It's an interesting read and seems to be relatively in-depth.
Reading through Section 2.7 of that report was a treat, thank you! (https://www.ntsb.gov/investigations/AccidentReports/Reports/...)
Excellent read thank you, that final section how the company struck off BRM (CRM) from the ships training prior was pretty damning.
Thank you for sharing. Sections 2.6 ("Captain's Decision-Making") and 2.7 are excellent case studies in leadership and authority -- especially the parts about "power distribution" and promoting a culture of open feedback.
The recommendations for the crew to "issue a direct command or to use a crew obligation statement — a statement in which the subordinate obligates the senior crewmember as part of the navigational team by using language such as 'we should' or 'our route'" is excellent. People who need to "manage up" at their jobs can start doing this today.
The recommendations for the crew to "issue a direct command or to use a crew obligation statement — a statement in which the subordinate obligates the senior crewmember as part of the navigational team by using language such as 'we should' or 'our route'" is excellent. People who need to "manage up" at their jobs can start doing this today.
One of my takeaways from the article is that the captain's confidence and willingness to do the right thing had been shaken after he'd been burned the last time he'd refused unsafe orders. I'm wondering if the 1st officer similarly faced bad consequences for doing the right thing and taking command -- mutiny charges are still a thing, aren't they?
That seemed like irresponsible speculation on Vanity Fair's part to me.
The root cause was a captain who mistakenly reacted off of 12 hour old data and ignored another source of data that contradicted it. Pretty surprising given the lengths the article goes through to impress upon the reader how "safety conscious" he was.
The root cause was a captain who mistakenly reacted off of 12 hour old data and ignored another source of data that contradicted it. Pretty surprising given the lengths the article goes through to impress upon the reader how "safety conscious" he was.
It is speculation, but it is also relevant. Commercial shipping may be uniquely prone to accidents caused by profit-driven decisions, as discussed in Perrow's Normal Accidents book.
The article probably emphasises the captain's safety consciousnes to prevent people from "blaming the operator", which is an instinctual and often wrong conclusion to jump to.
The NTSB report says: Performance. The chief mate's performance evaluations were consistently positive. His evaluation from June 2015 stated that he was “passionate” about his work and “an excellent instructor for the inexperienced.”
The article probably emphasises the captain's safety consciousnes to prevent people from "blaming the operator", which is an instinctual and often wrong conclusion to jump to.
The NTSB report says: Performance. The chief mate's performance evaluations were consistently positive. His evaluation from June 2015 stated that he was “passionate” about his work and “an excellent instructor for the inexperienced.”
In this case the blame is deserved though. The Coast Guard's report places the blame fully on the captain.
I don't believe it does place the blame fully on the captain, where did you see that?
The report conclusions mention several events and dozens of contributing factors.
If you want to think in terms of blame, which is not helpful to avoid future accidents nor for nurturing a safety culture, the report "blames" the ship owner, captain, crew, National Hurricane Center and the coast guard...
Some actions of the captain were a contributing factor, notably bridge resource management/communication.
The report conclusions mention several events and dozens of contributing factors.
If you want to think in terms of blame, which is not helpful to avoid future accidents nor for nurturing a safety culture, the report "blames" the ship owner, captain, crew, National Hurricane Center and the coast guard...
Some actions of the captain were a contributing factor, notably bridge resource management/communication.
"Coast Guard investigators virtually placed all of the blame on Michael Davidson, the El Faro's captain. Davidson underestimated the strength of the storm and the ship's ability to ride it out. He did not take enough measures to evade the storm, even though most of his crew raised concerns about the increase in strength of the storm. Investigators had stated that if Davidson had survived the storm, his actions would have been grounds for the Coast Guard to revoke his captain's license."
https://en.wikipedia.org/wiki/SS_El_Faro#US_Coast_Guard
https://en.wikipedia.org/wiki/SS_El_Faro#US_Coast_Guard
There's a movie that reviews that situation in a navy (nuclear submarine) context, it's called 'Crimson Tide'.
There’s an even older and probably more relevant (or at least similar) called “The Caine Mutiny.”
An lt takes command from the Captain during a typhoon due to the caltain’s actions/behavior and is subsequently tried for mutiny.
An lt takes command from the Captain during a typhoon due to the caltain’s actions/behavior and is subsequently tried for mutiny.
Not sure why you’re being downvoted, Jacques. We’ve been in a couple spats here but I’ve always respected your thoughts and opinion. Have (yet another) upvote :).
I suspect my comments in the thread about the Google shooting gained me a fresh batch of enemies.
If you look at the numbers for this site, these days, HN is too popular for it's own good, and it's getting really noisy.
On the front page right now, there are lots of garbage articles, with 500+ upvotes and comments, and any one of them handily enables new downvoters.
Way more people can downvote now, than ever before, and users relish new power. Having a respectful honor system, and some voting ring detection is great, but the truth is, there are a lot more shitty people on this site. What to do about that?
On the front page right now, there are lots of garbage articles, with 500+ upvotes and comments, and any one of them handily enables new downvoters.
Way more people can downvote now, than ever before, and users relish new power. Having a respectful honor system, and some voting ring detection is great, but the truth is, there are a lot more shitty people on this site. What to do about that?
I'm not saying you're wrong, but would you mind posting that from your non-greenbean account? I won't downvote you! It's just fake internet points anyway; why worry? I get downvoted every day I post. (Not every item I post.) Sometimes there is a lesson to be learned, sometimes there isn't.
[deleted]
[deleted]
"One of my takeaways from the article is that the captain's confidence and willingness to do the right thing had been shaken after he'd been burned the last time he'd refused unsafe orders."
This is just one serious problem in the maritime industry that I have first-hand knowledge of.
I loved sailing, but I don't do it any more because it's not the work that's dangerous, it's the other people.
This is just one serious problem in the maritime industry that I have first-hand knowledge of.
I loved sailing, but I don't do it any more because it's not the work that's dangerous, it's the other people.
Tell me about it. You'd think the ocean is a big enough place to avoid accidents, but somehow 16 year olds skippering their parents boats find their way into the path of my Whaler while I quietly sit fishing.
Also from the world of aviation, seems the captain had developed a classic case of get-there-itis. The symptoms are there, the prognosis is never good.
I could tell from the title this was going to be written by William Langewiesche. He really knows how to spin a good yarn.
All his articles and books seem to be movie-worthy. I'm not entirely convinced reality is that exciting, though.
All his articles and books seem to be movie-worthy. I'm not entirely convinced reality is that exciting, though.
The banter at the start reminded me of the start of Aliens. As transcript quotes, I assume it means James Cameron based it on reality.
The reality in this case was far, far more exciting than you ever want. If anything, the captain had a preternaturally professional calm.
> ship’s anemometer was in disrepair and had been for weeks
Can't measure wind; the co. didn't subscribe to hourly updates; a "software glitch" gave old reports.
The reality in this case was far, far more exciting than you ever want. If anything, the captain had a preternaturally professional calm.
> ship’s anemometer was in disrepair and had been for weeks
Can't measure wind; the co. didn't subscribe to hourly updates; a "software glitch" gave old reports.
That's a really interesting observation. I first saw Alien with my roommate soon after we had both graduated from Merchant Marine school and he laughingly made a similar comment: "typical sailor behavior."
And honestly, experienced Merchant officers have seen so much shit, it's hard to get them rattled.
And honestly, experienced Merchant officers have seen so much shit, it's hard to get them rattled.
A great example is his treatment of Air France 447: https://www.vanityfair.com/news/business/2014/10/air-france-...
Oh my, you weren't kidding:
On the last day of May in 2009, as night enveloped the airport in Rio de Janeiro, the 216 passengers waiting to board a flight to Paris could not have suspected that they would never see daylight again, or that many would sit strapped to their seats for another two years before being found dead in the darkness, 13,000 feet below the surface of the Atlantic Ocean. But that is what happened.
And I thought that name sounded familiar. William is the son of Wolfgang Langewiesche, author of one of the books that taught me how to fly: Stick and Rudder.
https://en.wikipedia.org/wiki/Stick_and_Rudder
On the last day of May in 2009, as night enveloped the airport in Rio de Janeiro, the 216 passengers waiting to board a flight to Paris could not have suspected that they would never see daylight again, or that many would sit strapped to their seats for another two years before being found dead in the darkness, 13,000 feet below the surface of the Atlantic Ocean. But that is what happened.
And I thought that name sounded familiar. William is the son of Wolfgang Langewiesche, author of one of the books that taught me how to fly: Stick and Rudder.
https://en.wikipedia.org/wiki/Stick_and_Rudder
Yep. S&R is one of the best nonfiction technical books ever, even if you have no particular interest in becoming a pilot.
A great article. I've read it a couple of times and this part always gets me:
"In the cockpit, the situation was off the scale of test flights. After Dubois arrived, the stall warning temporarily stopped, essentially because the angle of attack was so extreme that the system rejected the data as invalid. This led to a perverse reversal that lasted nearly to the impact: each time Bonin happened to lower the nose, rendering the angle of attack marginally less severe, the stall warning sounded again—a negative reinforcement that may have locked him into his pattern of pitching up, assuming he was hearing the stall warning at all."
They must have been so confused and frightened.
"In the cockpit, the situation was off the scale of test flights. After Dubois arrived, the stall warning temporarily stopped, essentially because the angle of attack was so extreme that the system rejected the data as invalid. This led to a perverse reversal that lasted nearly to the impact: each time Bonin happened to lower the nose, rendering the angle of attack marginally less severe, the stall warning sounded again—a negative reinforcement that may have locked him into his pattern of pitching up, assuming he was hearing the stall warning at all."
They must have been so confused and frightened.
His description of the "emotional inertia" that sets in is most terrifying of all. It really gives a feeling for the fear that grips. Instead of saying "they were afraid" he shows it. Good writing indeed.
If you like this article, I recommend finding his other pieces particularly about airplane crashes.
He writes in a gripping manner and normally seems to have done pretty thorough background research, as well as personal experience.
He writes in a gripping manner and normally seems to have done pretty thorough background research, as well as personal experience.
The El Faro sinking has also been covered extensively from a professional mariner perspective on gCaptain.
http://gcaptain.com/tag/el-faro-investigation/
http://gcaptain.com/tag/el-faro-investigation/
My big takeaway from all of this is that, given the choice between driving your boat into a hurricane and doing something else, something else should be the default.
Another conclusion was that a tiered subscription model that withholds life-saving information is a business I never want to be in. El Faro subscribed to the Bon Voyage System for weather updates, but they didn't subscribe to the routing guidance and always received six hour-old information. The stale data is a critical part of this story.
http://www.stormgeo.com/solutions/shipping/on-board-services...
http://www.stormgeo.com/solutions/shipping/on-board-services...
Another conclusion is to never be in a business that tries to save pennies by not subscribing to life critical data.
Another important part are the flashy graphics. People always trust you more with flashy graphics against a list of coordinates
As a professional curmudgeon, I have the same reaction to shiny graphics that I do to salesman I’m fancy suits: I’m not buying.
Charting data accurately is really really hard. On one team we had a guy who said charts are for asking questions, not making decisions, and I’ve found that to be a pretty safe default.
Charting data accurately is really really hard. On one team we had a guy who said charts are for asking questions, not making decisions, and I’ve found that to be a pretty safe default.
..and buggy software.
The stale data seemed to be partially a result of the software glitch. Therefore it's not clear at all that adding a "routing guidance" model on top of the same data would have been any better solution. The captain already established his own route given the same, largely wrong, data.
What they needed was hourly updates, which they had from the other service they subscribed to, which was ignored by the captain. Despite the staff being fully aware.
And the captain and staff should always be reading weather data from two sources anyway, for redundancy and accuracy.
These guys had plenty of information but over-relied on one which was 6-12hrs out-of-date at the worst time, which no indication it was old data.
These are solvable problems that have little to do with tiered pricing or a cheapness on part of the company. Both by the above redundancy and the weather company dating their data, and fixing software bugs for such critical software.
What they needed was hourly updates, which they had from the other service they subscribed to, which was ignored by the captain. Despite the staff being fully aware.
And the captain and staff should always be reading weather data from two sources anyway, for redundancy and accuracy.
These guys had plenty of information but over-relied on one which was 6-12hrs out-of-date at the worst time, which no indication it was old data.
These are solvable problems that have little to do with tiered pricing or a cheapness on part of the company. Both by the above redundancy and the weather company dating their data, and fixing software bugs for such critical software.
That's a pretty bad take away - no one wanted to drive trough the hurricane. The captain thought he was further away from the storm than he was, perhaps because of his use of an outdated weather feed.
A better take away might be, for important decisions, you should deeply understand the data(and importantly, it's limitations) you are using to make the decision.
A better take away might be, for important decisions, you should deeply understand the data(and importantly, it's limitations) you are using to make the decision.
The report seems to suggest that his decision to drive where he did was baffling, even with the information he had available. All his officers were telling him to do otherwise, multiple times.
He was looking at different data then they were looking at.
Why?
I mean, there seems to be no reasonable reason why that should be the case, he can see (and be notified) of everything they're seeing and he can substantiate his decision by pointing towards the data, and should do so - because in this case the other officers might have noticed problems with that data.
I mean, there seems to be no reasonable reason why that should be the case, he can see (and be notified) of everything they're seeing and he can substantiate his decision by pointing towards the data, and should do so - because in this case the other officers might have noticed problems with that data.
If that were all it was, then why are the investigators surprised by his decision-making?
"...you should deeply understand the data(and importantly, it's limitations) you are using to make the decision."
Very apt also for the Air France 447 crash this author wrote about.
Very apt also for the Air France 447 crash this author wrote about.
Another conclusion is to stick to land-based activities.
Or be able to go deep underwater when things get really nasty.
For a guy who grew up on a dryland wheat farm 400 miles from ocean, that is going in the wrong direction for me.
1. FYI the article is new but the disaster described happened on October 1, 2015.
2. I was astonished at this: > "It has been reported that a major merchant ship goes down somewhere in the world every two or three days; most are ships sailing under flags of convenience, with underpaid crews and poor safety records."
2. I was astonished at this: > "It has been reported that a major merchant ship goes down somewhere in the world every two or three days; most are ships sailing under flags of convenience, with underpaid crews and poor safety records."
As to point #2: https://www.fleetmon.com/maritime-news/?category=incidents has a sobering list.
The shipwreck rate staggerred me when I ran across it a few years back.
It's the key reason I don't find commercial marine nuclear propulsion viable.
It's the key reason I don't find commercial marine nuclear propulsion viable.
It still boggles my mind how three different GPS position formats can be used. And then not used correctly by search crews, which then missed location by 23nm.
https://i.imgur.com/nbwkCPY.png
https://i.imgur.com/nbwkCPY.png
Those formats are simply insane. It's well known that some people use decimal fractions of degrees, so why would anyone even consider using a decimal point to separate degrees and minutes?
Was this a big problem before computers? I doubt it. Back in the analog days two people would engage in a dialogue and ensure the units were agreed upon. Now people just blindly type data into a screen, or two computers blindly exchange packets without proper metadata.
It's ironic that we have 1000x better navigational accuracy than a 19th century tallship captain but we're much more careless with the data than he would have been.
It's ironic that we have 1000x better navigational accuracy than a 19th century tallship captain but we're much more careless with the data than he would have been.
I don't think it's carelessness, rather that 1000x more accurate navigation gives you a lot more things to care about. That 19th century captain measured the angle between the pole star and the local horizon with a sextant; the cartographers did it the same way, because that was the only thing you could do.
Today, we work with atomic clocks, and our instruments fabricate an angle between the z-axis and a 100th order spherical harmonic series that models the shape of the ocean surface. Every few years, we update the model to keep up with plate tectonics. (Not exactly, but you get the idea.) There are a lot more things to go wrong, and not everyone programs in sanity checks to prevent small corrections from causing large errors.
Today, we work with atomic clocks, and our instruments fabricate an angle between the z-axis and a 100th order spherical harmonic series that models the shape of the ocean surface. Every few years, we update the model to keep up with plate tectonics. (Not exactly, but you get the idea.) There are a lot more things to go wrong, and not everyone programs in sanity checks to prevent small corrections from causing large errors.
Fair point.
> That 19th century captain measured the angle between the pole star and the local horizon with a sextant;
Which immediately provided latitude. To get longitude, he did the same with just about any other star and correlated the angle to an accurate clock and carefully-prepared tables.
That "accurate clock" business was why longitude was such a difficult navigational problem for so long until John Harrison finally cracked it. (I know this is getting OT but I find it fascinating.)
https://en.m.wikipedia.org/wiki/John_Harrison
> That 19th century captain measured the angle between the pole star and the local horizon with a sextant;
Which immediately provided latitude. To get longitude, he did the same with just about any other star and correlated the angle to an accurate clock and carefully-prepared tables.
That "accurate clock" business was why longitude was such a difficult navigational problem for so long until John Harrison finally cracked it. (I know this is getting OT but I find it fascinating.)
https://en.m.wikipedia.org/wiki/John_Harrison
Yeah, coordinates are a pain in the ass, and here they're even using the same coordinates, just with different formatting. There's also UTM, deg-dec-min, deg-min-sec, and you have some differences between WGS84 and NAD27, and every group has their own preferences for their own reasons.
23 nanometers off, dang. If they'd only had a slightly wider field of view, they could have found those poor people. :(
Perhaps a little poor taste with the tone, but this is a great point. We're all talking about how poorly we handle units. Nm has two meanings, but I didn't even think about it until this comment.
Nautical Miles
The article mentions that there are major merchant vessels sinking every two/three days! That seems huge. How many deaths per year does that amount to? We're lucky to be in tech I guess.
Fermi estimate, about 120-180 sinkings a year * 30 crewmembers/ship * 0.75 chance of dying (I'm guessing some sink in port or are evacuated?) = 2700 deaths / year.
Whether by luck or not, that's actually not far off one of the better estimates I can find. The paper "Fatalities among the world's merchant seafarers (1990–1994)" [1] ends up with a rough estimate of 2200 fatalities per year worldwide for the early 1990s. This is based on relatively good data from 19 major maritime flags and major insurance underwriters, followed by more shaky extrapolation to the worldwide numbers. Of the 2200/yr, the authors attribute about half to maritime accidents, so the number due to actual sinkings and such is lower, around 1100/yr in their estimate. The other fatalities break down as: 1/4 of the total attributed to occupational accidents (falling off a ladder, etc.), and 1/4 to illness while at sea.
The paper does mention that other estimates have reached both considerably higher and considerably lower totals (they quote one author who estimated 13,000/yr in the 1980s!). It's hard to get good numbers because the worst safety records appear to be among flag-of-convenience nations who also have the worst reporting and insurance practices, so a large portion of total fatalities (probably) come from the places with the highest uncertainties. But even among rich nations they did still find it a comparatively dangerous occupation: it's safer to be a Danish-flagged seafarer than most other flags, but still >10x the fatality rate compared to having a job on land in Denmark.
[1] https://doi.org/10.1016/S0308-597X(98)00017-7
The paper does mention that other estimates have reached both considerably higher and considerably lower totals (they quote one author who estimated 13,000/yr in the 1980s!). It's hard to get good numbers because the worst safety records appear to be among flag-of-convenience nations who also have the worst reporting and insurance practices, so a large portion of total fatalities (probably) come from the places with the highest uncertainties. But even among rich nations they did still find it a comparatively dangerous occupation: it's safer to be a Danish-flagged seafarer than most other flags, but still >10x the fatality rate compared to having a job on land in Denmark.
[1] https://doi.org/10.1016/S0308-597X(98)00017-7
Why do you think the chance of dying on a sinking ship would be so incredibly high?
0.75 chance of dying is what you had on the Titanic!
Your number has no relation to reality at all.
https://en.m.wikipedia.org/wiki/List_of_shipwrecks_in_2017
0.75 chance of dying is what you had on the Titanic!
Your number has no relation to reality at all.
https://en.m.wikipedia.org/wiki/List_of_shipwrecks_in_2017
That is why I qualified it as a Fermi estimate. I was guessing randomly.
It's merely an estimate of order of magnitude.
Never ever underestimate how lucky we are to be in tech.
"Come work in tech, at least you won't get killed by hurricanes" sounds like a great endorsement.
I wonder what’s the survival rate of crew on a merchant vessel that’s sinking?
A recent (paywalled) article in the New Scientist stated that the fatal accident rate for ships' crews is twenty times that of the average British worker.
https://www.newscientist.com/article/mg23731623-300-ghost-sh...
https://www.newscientist.com/article/mg23731623-300-ghost-sh...
How does the fatal accident rate for, e.g., professional drivers compare with the average worker? I imagine crewing a ship is more dangerous, but humans are famously bad at evaluating risk.
So being crew on a merchant ship is more dangerous that cops (by far), and probably more dangerous on say a 10 year timeframe than being in the military (since you probably aren't at war that much over 10 years, unless you are unlucky).
For the British merchant fleet, 19.5 deaths per 100,000 seafarer-years [1].
For American loggers, 90 deaths per 100,000 worker-years [2] making it the most dangerous job.
For American taxi drivers, ~18 deaths (of which 8 are murders) per 100,000 full-time-worker-years [2]. Making taxi driver the most murdered profession - and it was worse around 2000, when the rate was an eye-watering 23.7 murders.
As you say, policing isn't the most dangerous profession; the rate there is 11 deaths of which 3.5 are murders.
[1] https://academic.oup.com/occmed/article/64/4/259/1464740 [2] https://www.washingtonpost.com/news/wonk/wp/2015/01/28/chart...
For American loggers, 90 deaths per 100,000 worker-years [2] making it the most dangerous job.
For American taxi drivers, ~18 deaths (of which 8 are murders) per 100,000 full-time-worker-years [2]. Making taxi driver the most murdered profession - and it was worse around 2000, when the rate was an eye-watering 23.7 murders.
As you say, policing isn't the most dangerous profession; the rate there is 11 deaths of which 3.5 are murders.
[1] https://academic.oup.com/occmed/article/64/4/259/1464740 [2] https://www.washingtonpost.com/news/wonk/wp/2015/01/28/chart...
Its probably including fishing which has I would imagine a much higher rate than crews on a cruise liner.
For anyone interested in stories like these, I highly recommend "Normal Accidents" by Charles Perrow - there's an entire chapter on maritime accidents.
Seconded, Normal Accidents is one of the best books I've ever read. In addition to the maritime accidents, there are chapters on nuclear plants, chemical plants, and dams. It is a great discussion of how modern industrial accidents rarely have a single cause but instead are cascade failures of systems whose complexity has evolved beyond what we can handle, and presaged many of the messes we've got into with hyper-scale software projects before those ever existed.
Is there a "sequel" to Normal Accidents about software? Because I'd buy that in a heartbeat.
Is there a "sequel" to Normal Accidents about software? Because I'd buy that in a heartbeat.
There's a book that seems similar to what you've described, called Atomic Accidents. It's a compelling narration of many notable mishaps in nuclear power, weapons,and research. Needless to say, it won't scratch the software itch.
Comp.risks has a lot of raw source material, and its moderator, Peter G. Neumann, has published a few books.
Ex-Goggler Yonatan Zunger has had some really good recent essays at, variously, Medium, Twitter, the Boston Globe, and Google+ (for which he was chief architect, and despite its general lack of widespread success, I consider his contributions and lessons-learned experience quite positively).
And he's a fan of Perrow.
Bruce Schneier's work is somewhat more focussed on crypto and security, but covers much this ground, especially later works.
Shoshana Zuboff also addresses social impacts of computers and data systems in her work.
Ex-Goggler Yonatan Zunger has had some really good recent essays at, variously, Medium, Twitter, the Boston Globe, and Google+ (for which he was chief architect, and despite its general lack of widespread success, I consider his contributions and lessons-learned experience quite positively).
And he's a fan of Perrow.
Bruce Schneier's work is somewhat more focussed on crypto and security, but covers much this ground, especially later works.
Shoshana Zuboff also addresses social impacts of computers and data systems in her work.
Check out stuff that Betrand Meyer, Eiffels dad, has written - he pretty much beat hell out of software mishaps to show how he'd tried to compensate for those issues in Eiffel. -- here's one of the best known -- https://archive.eiffel.com/doc/manuals/technology/contract/a...
Somehow related, "Command and Control" is an excellent read.
"Meltdown" by Clearfield and Tilcsik is another excellent book about accidents. The authors are Perrow disciples.
Is this the book where a nuclear technician drops a wrench from high up in a nuclear silo right onto the nuke and hears some horrifying sound (maybe a hissing?). I've been meaning to read that book, someone mentioned it in another HN post. Please let me know thanks.
Also "Atomic Accidents" (in the accident genre, but focused on early nuclear research accidents mostly). It's a surprisingly fun read for a topic where most of the people who are featured died horribly.
Wonder why shippers don't have a 24/7 Operations Center employing meteorologists and tracking weather systems and ships and in close contact with ship's crew. It struck me that there was a lot of friction for the crew to just plot up-to-date weather/forecasts and the ship's course. Crew was constantly fighting for current data and good/correct visualization.
Cost, most likely
I don't suppose anyone here knows whether this plays out differently with very large shippers? I could imagine that having a real 24/7 in-house operations center could be cost-effective and desirable for a giant line, especially one like Maersk that is both huge and somewhat reputation conscious. But I have no idea whether they do.
Depends on the line. Most big names do.
What amazed me reading the report was just how vulnerable the ship was to sinking if it had a list during a storm. While an 18 degree list is extreme it's not unforeseeable in a serious emergency. The problem was this put ventilation openings at a height where they were directly subject to green water infiltration. To make it worse only the intakes were 'watertight' the exhausts were only 'weathertight' meaning that this would have resulted in flooding that would have made the problem worse.
The real story here is that the ship was past it's retirement age AND poorly maintained. Any of the series of small events that happened due to the maintenance issues like the scuttle popping open, or the oil sumps not being filled all the way up (these guys were really macho idiots, how expensive is a sump top off really!). The engineers at the NTSB determined that they hit the sweet spot of their engines not working because of the extreme weather and list and unfilled sumps and broken anemometer so they couldn't tell the weather like sailors have been relying on for years. Basically tech is useless if you don't maintain it or replace it or you treat human beings like disposable parts to run your depreciating hulks. The whole world shipping community is running ridiculously old and poorly maintained wrecks sailed by the worlds poorest people for the world's richest assholes. How's that for a class struggle?
well there is a big gulf between storm and hurricane. if they had encountered a normal squall it is doubtful they would have been in trouble but they ran straight into a CAT3 hurricane which if you have experienced on land you know its not even remotely sane to be out in it.
True, but if you read the report even under non-insane circumstances the design of the ventilation system would have put the ship at risk in any gale or full gale situation. The main issue was the intakes were on the tween deck instead of the main deck and were outboard. Thus if the ship listed at all to the point where the tween deck was even partially submerged it risked serious flooding. This seems counter to the design of the ship where the tween deck was supposed to be open to the sea, but not awash certainly, but still in theory all fittings an openings on that deck should be able to be watertight. Indeed if you look at the report, the rest of the deck was.
>Fisker-Andersen wrote, “Captain Mike, diversion request heads up through Old Bahama Channel understood and authorized. Thank you for the heads up. Kind regards."
I'm not quite following that sentence. The article makes a lot of the "authorized", saying that meant the captain was being ordered to go directly to San Juan and not use the Old Bahama Channel. However, it seems to say that the diversion into the channel was authorized.
I'm not quite following that sentence. The article makes a lot of the "authorized", saying that meant the captain was being ordered to go directly to San Juan and not use the Old Bahama Channel. However, it seems to say that the diversion into the channel was authorized.
The difficulty of "authorized" was that a captain at sea should have full control to make safe navigational decisions without corporate, profit-minded authorization. Captain Mike's previous experience insinuated that a captain could override company orders but could expect to be quietly let go soon after.
Seems safe to assume those two tugboats cost a hell of a lot less than this accident and the wrongful death suits, not to mention the unnecessary fatalities.
Yeah, the trouble is though.. the tugboats got paid for and the steering gear didn't fail as it wasn't used.
Same here, he goes a different way and costs the company money and all of a sudden its not comparing the cost of the diversion with the cost of the loss of the ship and crew, its only comparing it to a safe but tense voyage.
Same here, he goes a different way and costs the company money and all of a sudden its not comparing the cost of the diversion with the cost of the loss of the ship and crew, its only comparing it to a safe but tense voyage.
captain: "Can i play it safe on the way back?"
boss: "Yes, authorized."
NTSB: "What do you mean authorized? It's not your decision. Did the captain ask you if he could go the safe way on the way out?"
boss: "Yes, authorized."
NTSB: "What do you mean authorized? It's not your decision. Did the captain ask you if he could go the safe way on the way out?"
I guess I still don't quite understand the importance of "authorized" since the captain wrote:
>Question I would like to transit the Old Bahama Channel on our return northbound leg
He was clearly asking, not informing.
>Question I would like to transit the Old Bahama Channel on our return northbound leg
He was clearly asking, not informing.
As another said, the Captain didn't specify what the question was.
Saying "authorized" was a big mistake. I work somewhere now where I have to be careful using that word because it implies you're taking on the responsibility, and if something goes wrong through no fault of your own, the blame now lies on you where as it wouldn't have if you just simply acknowledged receipt of the question/statement.
Stupid, but that's the world we live in.
Saying "authorized" was a big mistake. I work somewhere now where I have to be careful using that word because it implies you're taking on the responsibility, and if something goes wrong through no fault of your own, the blame now lies on you where as it wouldn't have if you just simply acknowledged receipt of the question/statement.
Stupid, but that's the world we live in.
That could be interpreted as "will it cause a schedule problem", or "is there another option I should consider" or "might this make it worse". Ultimately he never actually asks a question, just says what he's thinking about doing. To me it reads like the captain is keeping his superiors in the loop, but still has controll.
The problem with the response is that it has clearly been treated as a request ("diversion request heads up through Old Bahama Channel understood and authorized") that could have been denied.
It's obviously not an admission of guilt, but a red flag that's worth investigating.
The problem with the response is that it has clearly been treated as a request ("diversion request heads up through Old Bahama Channel understood and authorized") that could have been denied.
It's obviously not an admission of guilt, but a red flag that's worth investigating.
thanks... I'm finally getting all the bubbles in that thought smoothed down.
> He was clearly asking, not informing.
Right, that's the problem. It implies the captain was not free to make the decision himself (as he should have been).
Right, that's the problem. It implies the captain was not free to make the decision himself (as he should have been).
>Question I would like to transit the Old Bahama Channel on our return northbound leg to Jacksonville, Florida. This route adds an additional 160 nm to the route for a total of 1,261 nm. We will need to make around 21 knots for our scheduled 10/05 10:45 arrival time at Jacksonville pilot station.
oh, ok. So the presence of "authorized" in the response email brings up the question of routes in general being authorized by corporate. It isn't an order to go directly to San Juan.
Right--the authorization we saw was for the return trip, and the question was what that might have implied about the trip down.
The NTSB report notes that, earlier that year, during Tropical Storm Erika, the Captain felt no need to ask for authorization when changing course--he just made the change and notified HQ. This time, with his job on the line and future career in doubt, he seemed less willing to act independently.
The NTSB concluded that they had no solid evidence that the company was applying direct pressure for him to stick to the route, but they did note a whole host of implicit pressures, as well as a relative deprioritization of safety and crew management in the company culture as a whole, that might have adversely affected his decision-making.
The NTSB report notes that, earlier that year, during Tropical Storm Erika, the Captain felt no need to ask for authorization when changing course--he just made the change and notified HQ. This time, with his job on the line and future career in doubt, he seemed less willing to act independently.
The NTSB concluded that they had no solid evidence that the company was applying direct pressure for him to stick to the route, but they did note a whole host of implicit pressures, as well as a relative deprioritization of safety and crew management in the company culture as a whole, that might have adversely affected his decision-making.
See also an in-depth report from the AP: https://apnews.com/9d5904f83f6a4cf784828107fa7d038d
What’s really interesting about this is the sort of horror film aspect to these situations where everyone gets a bad feeling of impending doom but they push forward anyway.
People should trust that instinct. Takes a lot of effort to pull away from the brink, but you should do it. 100% of the time.
People should trust that instinct. Takes a lot of effort to pull away from the brink, but you should do it. 100% of the time.
The article pointed out that couple years earlier the captain did exactly as you suggest and pulled away from the brink because of safety concerns. His company punished him immediately with a big demotion.
It seems clear that the culture in the industry is set up to weed out those who want to err on the side of safety, and reward those who err on the side of maintaining their shipping schedules.
It seems clear that the culture in the industry is set up to weed out those who want to err on the side of safety, and reward those who err on the side of maintaining their shipping schedules.
Some don't recognize the situation is bad because of ignorance.
Some don't pull back because of cowardice.
The rest keep on going because of the sense of adventure. It makes you feel alive.
Some don't pull back because of cowardice.
The rest keep on going because of the sense of adventure. It makes you feel alive.
Agreed. Although, by changing the rout a couple of times they were pretty much following their instincts [alongside the weather reports].
That was hard to read.
Damn. That was intense. Thanks for posting this.
I suspect that human driven errors like this will lead the shipping industry to transition to autonomy. Through this whole story the autopilot was the only one steering the ship, the humans in the loop were responsible for plotting and course setting. Obviously this is a bad idea. It's not hard to imagine an expert software system that could have done a better job with this situation, especially in processing the text based weather data coming over the C-sat.
Apparently some Norweigans agree with me and are doing exactly this. I suspect this will become the future norm once the bugs are ironed out.
https://www.km.kongsberg.com/ks/web/nokbg0238.nsf/AllWeb/0CB...
Apparently some Norweigans agree with me and are doing exactly this. I suspect this will become the future norm once the bugs are ironed out.
https://www.km.kongsberg.com/ks/web/nokbg0238.nsf/AllWeb/0CB...
Anyone else get a weird pop-up after opening this article? I'm using Firefox for Android, and after I open the article and begin scrolling, a little box pops up that lists the names of 2 local financial institutions and the word "remove." As I scroll down, the box moves, too. If I tap on one of the names or the word "remove" the box disappears. I only get this on Firefox for Android.
The only reason it's rather worrying is because I happen to bank at both of the financial institutions on the list. They are both small, local institutions.
Tried opening the link on a couple other devices but was unable to recreate this. The pop-up appears every time I open the link in Firefox on my phone, though.
The only reason it's rather worrying is because I happen to bank at both of the financial institutions on the list. They are both small, local institutions.
Tried opening the link on a couple other devices but was unable to recreate this. The pop-up appears every time I open the link in Firefox on my phone, though.
Very nice and intense read, couldn’t stop till the end. I’m sure it would make a great movie. Pity for the people on board.
Ok, this article was the last straw. I've disabled javascript. It was impossible to read this article without it scrolling every few seconds for some ad script.
Wow. I know it is long, but read it if you have not. It’s very....human.
Turns out, the B.V.S reports were using raw data that was 10 to 12 hours old - and they didn't explicitly mention that. In the case of a rapidly evolving hurricane, it mattered a lot.
> The B.V.S. map included a time stamp that showed when the processing had been completed, but gave no indication of the age of the raw data on which the forecast was based. Davidson knew that all the forecasts were uncertain, and that they sometimes disagreed. But how aware was he that when he looked at the B.V.S. maps he was looking into the past?
[...]
> Davidson dismissed the plan with a thank-you and did not come to the bridge. Evidence suggests that he was still showing a preference for the animated B.V.S. graphics, which indicated the storm progressing more slowly.