Cisco devices must regenerate self-signed certificates before 2020(cisco.com)
cisco.com
Cisco devices must regenerate self-signed certificates before 2020
https://www.cisco.com/c/en/us/support/docs/field-notices/704/fn70489.html
66 comments
I think if someone lacks the awareness that hard-coding expiration dates is not a good idea, expecting them to think to this level is a bit much... :|
It's a good reminder nonetheless. I was on a team responsible for a web server farm and we noticed after a few years that the expiration date kept creeping closer to January since we would regularly kick off the process a few weeks before it actually expired, and decided one year to renew twice just to reset the clock away from January.
Why can't you be grateful for the gift of canceled vacations and overtime pay? Cisco cares - making sure you're home (in your data center) for the holidays.
x.509 certs all have a validity period. Of course they could have set them to 2099, but as far as I know you cannot change the validity without reissuing the cert.
It's called notAfter and since it's part of the signed document you cannot change it without invalidating the signature. In a custom system the issuer could (but probably shouldn't) sign a document which is identical save for the notAfter date, but in the Web PKI this would be a serious miss-issuance and have consequences.
By convention the notAfter field is filled out in Zulu timezone (ie UTC) and so these will expire simultaneously around the world.
By convention the notAfter field is filled out in Zulu timezone (ie UTC) and so these will expire simultaneously around the world.
The problem here isn’t that the certificates themselves have an expiration date, it’s that the code that generates them has a hard-coded date. Even if you try to generate a new certificate after 2020-01-01, the expiration date will still be 2020-01-01 on the new certificate.
[deleted]
What I find bizarre is software that accepts a self-signed certificate only to then reject the cert because it has expired. What security problem are they trying to solve there?
The reason to expiration dates on certificates is so that a compromised private key has a limited lifetime of applicability, under an assumption that you will be actively rotating keys and issuing new certificates that expire at later points in time. As far as I would think this consideration has nothing at all to do with whether the certificate is self-signed.
The model in most software that allows self-signed certs is that you eiter disable checking all together, or that the user accepts a specific cert.
If you disable checking all together, then an attacker doesn't need to compromise a private key. Any impersonation attack will just work.
If you require the user to accept a specific cert (and store that information) then you can just as well let the user accept an expired cert.
The current situation just creates a lot of hassle for people without any security benefit.
Key rotation is completely separate from the lifetime of a cert. You can renew certs while keeping keys constant forever. In practice, if your key gets compromised and you have to wait for the cert to expire, then you have a very big problem. Even with letsencrypt it is likely more than a month before the cert expires.
If you disable checking all together, then an attacker doesn't need to compromise a private key. Any impersonation attack will just work.
If you require the user to accept a specific cert (and store that information) then you can just as well let the user accept an expired cert.
The current situation just creates a lot of hassle for people without any security benefit.
Key rotation is completely separate from the lifetime of a cert. You can renew certs while keeping keys constant forever. In practice, if your key gets compromised and you have to wait for the cert to expire, then you have a very big problem. Even with letsencrypt it is likely more than a month before the cert expires.
> The current situation just creates a lot of hassle for people without any security benefit.
In my experience, many "security" people see this as the purpose of their job.
In my experience, many "security" people see this as the purpose of their job.
Also keeps CRLs as short as possible.
You can build good PKI using your own private CA just as well as using any publicly trusted root. Esp. in the realm of on-prem infra, that's not uncommon.
I think what Cisco is trying to solve is to give users the possibility to connect to the device over a secure connection in case a user has no access to a signed certificate by an authority.
So the user must still connect the self-signed certificate when connecting to the device (over ssh for example).
So in this case the user generates the certificate and also accepts it.
By the way, most ssh clients will tell you if a certificate changed. So the moment your self-signed certificate is compromised you will know.
So the user must still connect the self-signed certificate when connecting to the device (over ssh for example).
So in this case the user generates the certificate and also accepts it.
By the way, most ssh clients will tell you if a certificate changed. So the moment your self-signed certificate is compromised you will know.
> most ssh clients will tell you if a certificate changed
Most implementations of ssh in the wild don't use CA (or self) signed certificates. I dont know if Cisco even supports the use of ssh with certificates.
Most implementations of ssh in the wild don't use CA (or self) signed certificates. I dont know if Cisco even supports the use of ssh with certificates.
> I dont know if Cisco even supports the use of ssh with certificates.
They do -- in addition to public key auth, as GhettoMaestro mentioned.
They do -- in addition to public key auth, as GhettoMaestro mentioned.
Cisco SSH has supported public key auth for as far as I can recall.
SSH keys and certificates are different authentication mechanisms (I learnt this from https://news.ycombinator.com/item?id=20955465)
I would assume you can login via the CON (console) or the AUX port as a last resort.
Oh god. Thanks Cisco for dropping this 11 days before the end of the year. Wooo, glad I saw it here at least...
> Self-signed X.509 PKI certificates (SSC) that were generated on devices that run affected Cisco IOS® or Cisco IOS XE software releases expire on 2020-01-01 00:00:00 UTC. New self-signed certificates cannot be created on affected devices after 2020-01-01 00:00:00 UTC. Any service that relies on these self-signed certificates to establish or terminate a secure connection might not work after the certificate expires.
Certificate management is such a huge problem for big organizations. Does anyone have guidelines on how to do it properly? You really need a way to track who is using what and how they automatically get renewed a long time before expiry.
My view now is that certificates should be renewed very often - like every month, if you see something within 6 months of expiry there is a problem.
My view now is that certificates should be renewed very often - like every month, if you see something within 6 months of expiry there is a problem.
> You really need a way to track who is using what and how they automatically get renewed a long time before expiry.
If you have any sort of monitoring system in place, it should also be able to check for certificate expiration (or "validity" in general). Even the old antiquated nagios has had this ability since the dawn of time.
As a sort of last resort and/or where no other solution is in place, you can do what I did on a customer's network a few years ago. Fortunately, there wasn't much "security" in place (internally, at least).
nmap can easily 1) detect/identify TLS and 2) produce "grepable" output.
I simply gathered a list of all IP subnets that were in use and scanned all of it. Afterwards, it was fairly simple to go through the results and figure out which hosts were running services using TLS on which ports.
There are existing tools that can connect to services using TLS and gather the information from the presented certificate. If you can't find one that does what you need, that's no big deal, it really doesn't take much to write your own tool just for this purpose.
(In fact, something like this -- running on a host that has access to connect to all of your TLS services -- might be nice to have even if you have a "proper" solution in place. If (when?) something falls through the cracks, you'd have this as a sort of "fallback" or "backup" to alert you to anything your proper solution might have missed.)
If you have any sort of monitoring system in place, it should also be able to check for certificate expiration (or "validity" in general). Even the old antiquated nagios has had this ability since the dawn of time.
As a sort of last resort and/or where no other solution is in place, you can do what I did on a customer's network a few years ago. Fortunately, there wasn't much "security" in place (internally, at least).
nmap can easily 1) detect/identify TLS and 2) produce "grepable" output.
I simply gathered a list of all IP subnets that were in use and scanned all of it. Afterwards, it was fairly simple to go through the results and figure out which hosts were running services using TLS on which ports.
There are existing tools that can connect to services using TLS and gather the information from the presented certificate. If you can't find one that does what you need, that's no big deal, it really doesn't take much to write your own tool just for this purpose.
(In fact, something like this -- running on a host that has access to connect to all of your TLS services -- might be nice to have even if you have a "proper" solution in place. If (when?) something falls through the cracks, you'd have this as a sort of "fallback" or "backup" to alert you to anything your proper solution might have missed.)
I'd like to know as well. At my work we had to partner with a major bank and they demanded a fully signed (sigh) client certificate for the integration. The way their API works means that we have to do a planned hard-cutover before the certificate expires. There's a 60% chance that we'll have trouble reaching out to the right people when the time comes.
Ugh.
Ugh.
This is a common problem in healthcare insurance. API connections will stop working for about 3 weeks every year as each side performs a complicated dance of noticing the expiration and updating the systems to accept a new public certificate.
IMHO, 60% sounds low...
A short period of renewal would mitigate the problem where every employee on the notification list has become a former employee / invalid email by the time the actual expiry occurs.
That kind of turnover seems crazy. You should also be using an email distribution list as the "notification target" which is kept up to date when someone joins or leaves the team...
We take a belt-and-suspenders approach, mainly because I have a huge fear of the embarrassment of letting a cert (or domain name) expire.
We built a DB to track them; nothing special, it sends email reminders. Been meaning to just publish calendar reminders, but haven't gotten to it.
We also check expiry dates on the live certificates via one of our monitoring systems (Zabbix, but this isn't a built-in check) that will start yelling, I think, 2 weeks before expiry.
We built a DB to track them; nothing special, it sends email reminders. Been meaning to just publish calendar reminders, but haven't gotten to it.
We also check expiry dates on the live certificates via one of our monitoring systems (Zabbix, but this isn't a built-in check) that will start yelling, I think, 2 weeks before expiry.
You can use something like Vault with the PKI backend and an audit engine set up. You will get a full trace of all certs delivered and an API endpoint to update them.
You should know your assets. If you know your assets, it's pretty trivial to write a script or program that checks when their certificate expires, sticks it in a database, and alerts you when it's getting close to time to renew. Of course, if you don't know your assets, you hopefully at least know your IP space and can scan that.
Fully automated issuance and rotation on the order of hours. Use stuff like SPIFFE [0]. Won't help you much with 3rd-party proprietary hardware but you can at least avoid dealing with long-lived certificate management for your own software.
[0] https://spiffe.io/
[0] https://spiffe.io/
There is software like Venifi, but it requires everyone to follow a specific process. You still have outliers.
check Venafi for scanning your network for certs and managing them
you can also build scanner yourself on top of nmap and store discovered certs in one DB and link it with your configuration management db
you can also build scanner yourself on top of nmap and store discovered certs in one DB and link it with your configuration management db
Automate every device separately. I've written browser pupetteering scripts for that.
Just had to emergency deal with this issue within a large scale complex ISP env. Fun times!
Edit: And yes as others pointed out, right into the day or two before holiday moratorium.
Edit: And yes as others pointed out, right into the day or two before holiday moratorium.
I'm surprised there's no workaround "set the clock back to 2019, generate a certificate, then fix the clock time". Would that not work?
The generated certificate would also expire in 2020, so it would be invalid immediately.
Correct me if I’m wrong, but is this just another measure by Cisco to block off the second hand market for the equipment?
Doubtful. Cisco software code quality isn't the best, even in their specialist fields (see the number of BGP related bugs that are patched every year, still). This looks like just another stupid bug that has already been fixed in a future release.
Yup. They have lots of smart people, but something has been badly wrong on the code quality front for decades. I remember not long after the turn of the century doing inter-continental live debug sessions with their team because the multicast IPv6 acceleration (MLD snooping, like IGMP snooping but for IPv6) they'd shipped was completely broken in our Catalyst 3750s. As usual the authorised workaround was to disable acceleration, destroying the value added by Cisco's incredibly expensive product so I was on Transatlantic conference calls after hours trying to get them to reproduce the issue so they could actually ship us a fix.
My theory is that the business is more focused on getting contracts and sales, and getting in some useless magic quadrant than they are on actually providing a quality product.
You can just get a cert (from your corporate CA, or a public one, or just some one you spin up for the purpose) to fix this issue.
It's very likely just somebody harcoded a date "in the distant future" in some OpenSSL config file or script that generates the self-signed cert.
It's very likely just somebody harcoded a date "in the distant future" in some OpenSSL config file or script that generates the self-signed cert.
Care to explain how you think this issue is even remotely related to the grey market for Cisco products?
I'm having trouble trying to come up with any link between the two.
---
In my opinion, with the exception of perhaps very small networks, no one who's "doing it right" should really be affected by this issue.
I can kinda maybe understand folks using self-signed certificates for VPNs but, really, if you're doing much at all with certificates you should either be running your own internal PKI and/or getting certificates by somewhere else.
My previous employer had literally hundreds of devices using their own self-signed certificates (for device management) on them. Many of them were the "default" certificates that shipped on the devices, so it was the same certificate shared across hundreds of devices. As someone who gives a shit about security, that drove me absolutely nuts but no one else gave a damn. (Even worse, since nearly all of these devices used the same credentials, you'd only need to MITM one of them in order to "pwn" damn near everything on the entire network!)
I'm having trouble trying to come up with any link between the two.
---
In my opinion, with the exception of perhaps very small networks, no one who's "doing it right" should really be affected by this issue.
I can kinda maybe understand folks using self-signed certificates for VPNs but, really, if you're doing much at all with certificates you should either be running your own internal PKI and/or getting certificates by somewhere else.
My previous employer had literally hundreds of devices using their own self-signed certificates (for device management) on them. Many of them were the "default" certificates that shipped on the devices, so it was the same certificate shared across hundreds of devices. As someone who gives a shit about security, that drove me absolutely nuts but no one else gave a damn. (Even worse, since nearly all of these devices used the same credentials, you'd only need to MITM one of them in order to "pwn" damn near everything on the entire network!)
The certificates on the Cisco WLC only lasted 10 years, you have to upgrade the WLC and all the access points to a new version that includes a command to turn off the certificate check. If you want to upgrade an AP after that date you need to set back the clock first, so much fun
They seem to be highlighting SIP/VOIP as the main issue. And downplaying ssh access into the router based on an assumption that most don't use x509 certs for ssh. Does that sound right?
Also, you aren't affected if you actually have a CA. This only blows up if you use the Cisco device's own self-signed certs which it always mints (in affected versions of IOS) with notAfter 2020-01-01.
For stuff like web tools that might happen by least effort, but for SSH it's a bunch of extra effort, 90% of the effort of doing it properly, yet with almost none of the security benefits. Anybody who cares will use their own CA and be unaffected, anybody who doesn't care never enabled certs for SSH on the Cisco boxes.
For stuff like web tools that might happen by least effort, but for SSH it's a bunch of extra effort, 90% of the effort of doing it properly, yet with almost none of the security benefits. Anybody who cares will use their own CA and be unaffected, anybody who doesn't care never enabled certs for SSH on the Cisco boxes.
Yes, most don't use X.509 for SSH (note: public key authentication is not the same thing).
Most SSH access doesn't use x.509, so yes that sounds right.
I still remember the time when 1st of January 2020 was so far in the future that it seemed it will never happen.
I did data recovery on an external drive in 2011. It was pretty bad but I only needed a dozen files or so, rest was backed up. One of the files restored had a borked creation time in May 2020. For some reason I never fixed that, so whenever I accessed that folder and sort by date, that file would be at the top. This is will stop happening soon. I grew so fond of this quirk that I'm almost sad.
touch -d 'Jan 01 2030'
We keep telling us the same about https://en.wikipedia.org/wiki/Year_2038_problem
I've been booking meetings for 2020 and it still looks weird.
It's probably going to take me about 2 months after the new year stop writing/entering "2019" on forms. And checks. And anything else that calls for a date.
Pretty much the same thing happens to me every year.
Pretty much the same thing happens to me every year.
I recently decided to switched to a 36 hour week. I accidentally said I wanted the switch to start on January first 2019 rather than 2020...
Things seem to have been fixed, but it was a scary mistake to make.
Things seem to have been fixed, but it was a scary mistake to make.
[deleted]
I guess that what happens when we get old. We're going to have to push back that shiny future to 2050 (at least).
This serves as a good reminder of why it’s a bad idea to set expiration dates at the end of a year, where lots of people are on leave for the holidays (this notice came out only 2 days ago). Instead try sometime less complicated, like mid March or Sept.