Maryland hasn't updated Covid case data for 15 days due to a security incident(health1.maryland.gov)
health1.maryland.gov
Maryland hasn't updated Covid case data for 15 days due to a security incident
https://health1.maryland.gov:443/incidentupdate/Pages/default.aspx
62 comments
This is actually pretty bad, their mobile app wasn't updated to explain the situation and was showing 0 new cases for the timeframe. I personally know people who took that at face value.
Looks like we hugged the server to death? Won't load for me or archive.is.
I feel sorry for their admins. First they get hacked and then they get DDoS'd.
Sorry, it was up when I posted this. Here is a shorter version (without a FAQ): https://health.maryland.gov/newsroom/Pages/Maryland-Departme..., here is their dashboard showing no updates since Dec. 3: https://coronavirus.maryland.gov/, and here is a news story: https://www.youtube.com/watch?v=cLTaIW24fxU.
Maryland Governor Larry Hogan tested positive today: https://twitter.com/GovLarryHogan/status/1472946270507397125
So did Elizabeth Warren, what's your point?
I just wanted to reference additional news pertaining to covid in Maryland.
...until today: https://coronavirus.maryland.gov/ now has data through 12/19, with daily updates to resume tomorrow per an update on that same page.
In the last 100 days, the MD Department of Information Technology (DoIT) has submitted $3,000,000 of ____emergency____ procurement requests for network monitoring, intrusion detection, and firewall assistance.
All have been approved and funded.
Sept 2021:
- Splunk Emergency Professional Services - $800k
- src: https://bpw.maryland.gov/MeetingDocs/2021-Sept-1-Agenda.pdf
Dec 2021:
- Splunk Emergency Professional Services - $1.6m
- Palo Alto Emergency Professional Services - $360k
- Ciena Emergency Professional Services - $240k
- src: https://bpw.maryland.gov/MeetingDocs/2021-Dec-1-Agenda.pdf
So...what do you think is going on at Maryland's Department of IT?
____TRULY SHAMEFUL INCIDENT COMMUNICATIONS____
Bad things happen in IT departments every day. Some can be anticipated, some even possibly prevented - but there will always exist the risk of data breach/attack.
“Public officials have no higher responsibility than keeping the American people safe, and there is no greater threat to their safety than the cyber vulnerabilities of the systems that support our daily lives,” said Governor Hogan in November 2021.
Secretary Leahy's commitment to the Governor and the people of Maryland is to lead the state's IT resources in taking all reasonable measures to protect against such risks.
I do not yet know if this situation could have been reasonably prevented or if it is due to MD DoIT negligence/oversight. The flurry of emergency procurement requests suggests awareness and good intentions. And yet - here we are...
What is clear to me, however, is that Secretary Leahy and his directs have ___failed to effectively manage communications___ throughout the response and remediation effort.
Most damning, in my opinion, is the use of the incident report as a venue for high-fiving and taking victory laps:
> Because of the state’s aggressive cybersecurity strategy, and the use of MD THINK and other cloud-based services, many of the department’s core functions were not affected.
> src: https://health.maryland.gov/incidentupdate/Pages/default.asp...
Are you serious? It's been over two weeks. And it's clear from your report that you're nowhere close to understanding what happened, let alone fixing it.
____FUNDAMENTAL MISUNDERSTANDING OF INCIDENT IMPACT____
Whoever approved this for publication is oblivious to the downstream, rippling effects this incident continues to have:
- At the time of writing, MD's local public health departments have been ___fighting a war blindfolded___ for 16 days.
- The impact has rippled ___beyond MD's state borders___, impacting CDC data feeds as well. Hospitalizations (which are reported by each hospital network directly to the CDC) are trending upward while new cases (reported by each state's IIS to the CDC) have flatlined: https://imgur.com/gallery/k2gG5GS.
____AND WHAT THE HECK IS MD THINK?____
MD Think is Maryland's cloud application platform. The state, along with Deloitte and an army of subcontractors, has been working on migrating legacy applications to this new infrastructure since 2017.
The report points to the success of MD THINK in isolating the impact away from core MD Health Department services.
Is MD Think really a success? Is this a case of what happens when legacy applications are left outside to rot in the sun?
No.
It turns out that MD Think has been a nightmare since it's inception. A series of program failures that jeopardized open enrollment for medicaid recipients led to a full audit of the MD DoIT in 2020.
Here is what they had to say about MD THINK:
> "...for one project (MD THINK), the estimated $314.1 million cost of completion had increased by $141.1 million (81.5 percent)...an explanation was not provided for the increase..."
And on the topic of security:
> "We identified 55 firewall rules that allowed traffic from any source to 71 unique network destinations as well as a small network segment within DoIT’s internal network without IDPS coverage..."
> "...access to personally identifiable information (PII) for State vendors stored in the State’s Financial Management Information System (FMIS) was not adequately restricted, and the PII was accessible to thousands of State employees."
> src: https://www.ola.state.md.us/umbraco/Api/ReportFile/GetReport...
____What About ImmuNET?____
A question that begs an answer is why ImmuNet, the state's immunization registry, is not included under this "titanium umbrella of protection?"
And while there is a ton of publicly available paperwork about the scope of MD THINK, including the RFP, Deloitte's response, and transcripts of hearings to explore the audit report's findings, ImmuNet is not mentioned. Not once.
Please keep in mind - ImmuNet is not some podunk, state government hobby project. It is __law__ that Maryland retain vaccination records for its residents. School systems, employers, health providers, and the CDC all rely on Immunization Information Systems (IIS) for ground truth vaccination records. It is the public health equivalent of DMV.
____PASSION DISCLAIMER____
The lion's share of my time this year has been spent working with state and local COVID vaccination programs within upstate New York. I have seen firsthand how important accurate, timely test and vaccination data is to how local public health departments coordinate response.
It was infuriating to see the MD DoIT downplay disruptions to COVID related data feeds. We are going into year three of a war that demands accurate, timely data to inform decision making. What the hell have you guys been doing for two weeks?!
The country's public health departments are filled with the true unsung heroes of the pandemic. They're underpaid, exhausted, afraid, and constantly under attack - yet they're our last (and in many ways, only) line of defense.
SO...
To the three people that made it all the way to the end of this post - I ask just one thing:
IF YOU EVER FIND YOURSELF INVOLVED IN A FUCKUP THAT IMPACTS THE DEPARTMENT OF PUBLIC HEALTH:
- RULE 1: Apologize and fix it. Now. With the kindness, urgency, and empathy a mission controller would show a stranded astronaut--Proceed to RULE 6.
- RULE 2: If the problem is not your fault--return to RULE 1.
- RULE 3: If the problem isn't even real--return to RULE 1.
- RULE 4: If they are blaming you for a problem they created on their own--return to RULE 1.
- RULE 5: If the problem is related to Microsoft Access, FoxPro, or any other technology you swore an oath to avoid--return to RULE 1.
- RULE 6: The final rule. Do not downplay the importance of data and tools public health resources rely upon. And NEVER, EVER respond to the problem by boasting about all the great technical achievements made everywhere BUT the underfunded, under-appreciated public health department.
All have been approved and funded.
Sept 2021:
- Splunk Emergency Professional Services - $800k
- src: https://bpw.maryland.gov/MeetingDocs/2021-Sept-1-Agenda.pdf
Dec 2021:
- Splunk Emergency Professional Services - $1.6m
- Palo Alto Emergency Professional Services - $360k
- Ciena Emergency Professional Services - $240k
- src: https://bpw.maryland.gov/MeetingDocs/2021-Dec-1-Agenda.pdf
So...what do you think is going on at Maryland's Department of IT?
____TRULY SHAMEFUL INCIDENT COMMUNICATIONS____
Bad things happen in IT departments every day. Some can be anticipated, some even possibly prevented - but there will always exist the risk of data breach/attack.
“Public officials have no higher responsibility than keeping the American people safe, and there is no greater threat to their safety than the cyber vulnerabilities of the systems that support our daily lives,” said Governor Hogan in November 2021.
Secretary Leahy's commitment to the Governor and the people of Maryland is to lead the state's IT resources in taking all reasonable measures to protect against such risks.
I do not yet know if this situation could have been reasonably prevented or if it is due to MD DoIT negligence/oversight. The flurry of emergency procurement requests suggests awareness and good intentions. And yet - here we are...
What is clear to me, however, is that Secretary Leahy and his directs have ___failed to effectively manage communications___ throughout the response and remediation effort.
Most damning, in my opinion, is the use of the incident report as a venue for high-fiving and taking victory laps:
> Because of the state’s aggressive cybersecurity strategy, and the use of MD THINK and other cloud-based services, many of the department’s core functions were not affected.
> src: https://health.maryland.gov/incidentupdate/Pages/default.asp...
Are you serious? It's been over two weeks. And it's clear from your report that you're nowhere close to understanding what happened, let alone fixing it.
____FUNDAMENTAL MISUNDERSTANDING OF INCIDENT IMPACT____
Whoever approved this for publication is oblivious to the downstream, rippling effects this incident continues to have:
- At the time of writing, MD's local public health departments have been ___fighting a war blindfolded___ for 16 days.
- The impact has rippled ___beyond MD's state borders___, impacting CDC data feeds as well. Hospitalizations (which are reported by each hospital network directly to the CDC) are trending upward while new cases (reported by each state's IIS to the CDC) have flatlined: https://imgur.com/gallery/k2gG5GS.
____AND WHAT THE HECK IS MD THINK?____
MD Think is Maryland's cloud application platform. The state, along with Deloitte and an army of subcontractors, has been working on migrating legacy applications to this new infrastructure since 2017.
The report points to the success of MD THINK in isolating the impact away from core MD Health Department services.
Is MD Think really a success? Is this a case of what happens when legacy applications are left outside to rot in the sun?
No.
It turns out that MD Think has been a nightmare since it's inception. A series of program failures that jeopardized open enrollment for medicaid recipients led to a full audit of the MD DoIT in 2020.
Here is what they had to say about MD THINK:
> "...for one project (MD THINK), the estimated $314.1 million cost of completion had increased by $141.1 million (81.5 percent)...an explanation was not provided for the increase..."
And on the topic of security:
> "We identified 55 firewall rules that allowed traffic from any source to 71 unique network destinations as well as a small network segment within DoIT’s internal network without IDPS coverage..."
> "...access to personally identifiable information (PII) for State vendors stored in the State’s Financial Management Information System (FMIS) was not adequately restricted, and the PII was accessible to thousands of State employees."
> src: https://www.ola.state.md.us/umbraco/Api/ReportFile/GetReport...
____What About ImmuNET?____
A question that begs an answer is why ImmuNet, the state's immunization registry, is not included under this "titanium umbrella of protection?"
And while there is a ton of publicly available paperwork about the scope of MD THINK, including the RFP, Deloitte's response, and transcripts of hearings to explore the audit report's findings, ImmuNet is not mentioned. Not once.
Please keep in mind - ImmuNet is not some podunk, state government hobby project. It is __law__ that Maryland retain vaccination records for its residents. School systems, employers, health providers, and the CDC all rely on Immunization Information Systems (IIS) for ground truth vaccination records. It is the public health equivalent of DMV.
____PASSION DISCLAIMER____
The lion's share of my time this year has been spent working with state and local COVID vaccination programs within upstate New York. I have seen firsthand how important accurate, timely test and vaccination data is to how local public health departments coordinate response.
It was infuriating to see the MD DoIT downplay disruptions to COVID related data feeds. We are going into year three of a war that demands accurate, timely data to inform decision making. What the hell have you guys been doing for two weeks?!
The country's public health departments are filled with the true unsung heroes of the pandemic. They're underpaid, exhausted, afraid, and constantly under attack - yet they're our last (and in many ways, only) line of defense.
SO...
To the three people that made it all the way to the end of this post - I ask just one thing:
IF YOU EVER FIND YOURSELF INVOLVED IN A FUCKUP THAT IMPACTS THE DEPARTMENT OF PUBLIC HEALTH:
- RULE 1: Apologize and fix it. Now. With the kindness, urgency, and empathy a mission controller would show a stranded astronaut--Proceed to RULE 6.
- RULE 2: If the problem is not your fault--return to RULE 1.
- RULE 3: If the problem isn't even real--return to RULE 1.
- RULE 4: If they are blaming you for a problem they created on their own--return to RULE 1.
- RULE 5: If the problem is related to Microsoft Access, FoxPro, or any other technology you swore an oath to avoid--return to RULE 1.
- RULE 6: The final rule. Do not downplay the importance of data and tools public health resources rely upon. And NEVER, EVER respond to the problem by boasting about all the great technical achievements made everywhere BUT the underfunded, under-appreciated public health department.
Thank you for posting this. As a Maryland resident I found this really enlightening. I wasn't aware of this and I'm pretty curious now what's going on. It makes me wonder how or if I can help, as an ordinary citizen.
I read through those procurement requests. The reasons they cite in the remarks are interesting.
Sept 2021, Splunk:
> The staff providing the Splunk support services resigned as of March 31, 2021, and the contractor has been unable to replace them. The Department requested these support services through another contract vehicle, but that contractor was unable to source qualified and competent candidates in the appropriate labor categories and with the rates provided in the contract.
Dec 2021, Palo Alto Firewalls:
> Although the Department has two contract vehicles available to staff this need, neither contractor has been successful in staffing the role. One contractor who once provided the services lost three of their four resources in the past 12 months. The other contractor has been unable to source a candidate with the requisite skills and experience in the appropriate labor categories and rates provided in the contract.
So people are quitting, and they can't find replacements. Wow, that makes me think we're seeing effects from the "Great Resignation". I wonder what the "rates provided in the contract" are.
I read through those procurement requests. The reasons they cite in the remarks are interesting.
Sept 2021, Splunk:
> The staff providing the Splunk support services resigned as of March 31, 2021, and the contractor has been unable to replace them. The Department requested these support services through another contract vehicle, but that contractor was unable to source qualified and competent candidates in the appropriate labor categories and with the rates provided in the contract.
Dec 2021, Palo Alto Firewalls:
> Although the Department has two contract vehicles available to staff this need, neither contractor has been successful in staffing the role. One contractor who once provided the services lost three of their four resources in the past 12 months. The other contractor has been unable to source a candidate with the requisite skills and experience in the appropriate labor categories and rates provided in the contract.
So people are quitting, and they can't find replacements. Wow, that makes me think we're seeing effects from the "Great Resignation". I wonder what the "rates provided in the contract" are.
Digital ID/health seems dead in the water.
Reichhardt(4)
covid isn't going to reach some new all time high infection rate because one monitoring system in one state was down for a few weeks (if that's the sentiment here. article doesn't load)
Counting cases and advertising them every time the news shows start is useless, it just fuels the paranoia.
This virus won't just go away in a few months, we'll have to live with it, just like Humans have done with every other virus since forever.
This virus won't just go away in a few months, we'll have to live with it, just like Humans have done with every other virus since forever.
> This virus won't just go away in a few months, we'll have to live with it, just like Humans have done with every other virus since forever.
This statement mean different things to different people. Do we live with AIDS with or without condoms ? Do we live with Covid with or without masks and vaccines and other prophylactic measures ?
This statement mean different things to different people. Do we live with AIDS with or without condoms ? Do we live with Covid with or without masks and vaccines and other prophylactic measures ?
are you suggesting we keep wearing masks in public forever? Surely someone takes this stance but it sounds ludicrous to me. (Although I wouldn't mind normalizing it for people who are feeling under the weather but still must travel, say.)
Maybe there should be mandatory when numbers are too high or maybe we should try to educate people to wear a mask when sick. Like it seems they do it in Asia.
I am glad we have FFP2/n95 options.
I am glad we have FFP2/n95 options.
tasogare(1)
Wow, definitely, the caseload matters a lot. 'Flatten the curve' has always been a very real policy, if you watch the reports from State/Provincial agencies you can very clearly seem them watch the number of available beds, and particularly how much COVID caseload affects other hospital operations i.e. delayed surgeries etc.. This definitely informs policy.
It also helps motivate people to get vaccinated, with a 3rd shoot booster seemingly providing decent protection against Omicron, it's something we probably want to get people to.
Humans survive for millennia because we react rationality to situations, not because we just lay down and die.
It also helps motivate people to get vaccinated, with a 3rd shoot booster seemingly providing decent protection against Omicron, it's something we probably want to get people to.
Humans survive for millennia because we react rationality to situations, not because we just lay down and die.
>and particularly how much COVID caseload affects other hospital operations i.e. delayed surgeries etc..
And what is more important, the people with deadlier diseases who didn't get surgeries or treatments that were postponed, the screanings that weren't done that could allow an early treatment of someone who is sick too, to prevent other persons that could potentially become sick?
>Humans survive for millennia because we react rationality to situations, not because we just lay down and die.
Read again, you totally missed my point.
And what is more important, the people with deadlier diseases who didn't get surgeries or treatments that were postponed, the screanings that weren't done that could allow an early treatment of someone who is sick too, to prevent other persons that could potentially become sick?
>Humans survive for millennia because we react rationality to situations, not because we just lay down and die.
Read again, you totally missed my point.
> And what is more important, the people with deadlier diseases who didn't get surgeries or treatments that were postponed, the screanings that weren't done that could allow an early treatment of someone who is sick too, to prevent other persons that could potentially become sick?
> Read again, you totally missed my point.
I read this comment a few times, but the sentence structure made it difficult to understand what you're trying to say. I'm entirely unsure which of the aforementioned situations you think is more desirable. If you're trying to make a point, instead of asking people to read again, you may want to consider rewriting your points to be more clear.
> Read again, you totally missed my point.
I read this comment a few times, but the sentence structure made it difficult to understand what you're trying to say. I'm entirely unsure which of the aforementioned situations you think is more desirable. If you're trying to make a point, instead of asking people to read again, you may want to consider rewriting your points to be more clear.
humans are definitely not rational. Our reaction has been driven by fear and anxiety out of all proportion to the actual threat here. Obesity is still a bigger killer than covid but we have not spent the same resources tackling it.
Well in the age of social media where people love to get their daily doom/rage porn dopamine hit, it's a big money maker for media companies
Not quite every other virus. We eliminated a few handful of diseases.
(And of course, lots and lots of viruses were eliminated naturally. Just by natural selection and competition. There are even some hints that coronaviruses are replacing influenca.)
(And of course, lots and lots of viruses were eliminated naturally. Just by natural selection and competition. There are even some hints that coronaviruses are replacing influenca.)