Sounds about right. And not the first time it happens either. I recall getting a few of those instant unit 3 panic over the past few years with Ubuntu. Often with things not as common out there in production, like tc (which in our case we were using in production to work around conntrack race conditions), and sometimes we also got non-panicking but absolutely production/nerve wrecking issues like TCP window size calculation overflows after the window went to zero due to a temporary slow consumer - freezing the window size to a few bytes only instead of getting a prompt full window recovery.
Not to mention we’ve also had our fair share of production triple faults from bugs in the Intel firmware patches for Spectre, which took weeks to investigate & fix between ourselves struggling to keep our exchange up & running, Intel, and AWS.
And that is why there’s value in the CoreOS/ContainerLinux-like solutions we designed & implemented nearly a decade ago now. Being able to promptly rollback any kernel/system/package upgrades at once - either manually or either after it’s detected a few panics in quick successions is actually quite awesome. Not to mention the slow update rollout strategy baked into the Omaha controller.
But the reality is that the what-ifs are always the hardest to market, nearly always after-thoughts and with fast-spiking/fast-decaying traction after major events.
Back in 2014, my once Chinese girlfriend suffered from Systemic lupus erythematosus (SLE), was first diagnosed when she was 12, then prescribed by a Chinese hospital a mixture of drugs to take every day without interruption - which appeared to have rendered the disease asleep. Fast forward to her 24th birthday and the disease re-surfaced dramatically. We promptly went to the University of Tours' hospital and had the opportunity to meet with some of the finest rare disease professors.. only to be told that her continued use of the drugs she was prescribed back then actually had to be reduced/stopped progressively as the disease was being controlled.. Instead, the continuous prescriptions significantly damaged her kidneys, and made her sick thyroid dependent and accustomed to the drugs, allowing the disease to come back in full force, in a much more aggressive fashion than before and that it would be now hard to combat. By her 25th birthday, and while we had been able to share unique times of happiness that year during remission phases - flares would make sitting up on her bed and buttoning her shirt nearly impossible due to the constant joint pains / stiffness, fatigue, etc - and her immune system would often turn against her own organs.. eventually pivoting to her brain, causing severe confusion and depression. Shortly thereafter, she took her own life to put an end her own sufferings.
It deeply pains me to witness the US & Chinese health care systems, such tremendous powers with staggering economies, being nowhere near the quality of the free healthcare provided by France - to French citizens and temporary residents alike.
Organizations where the big $ is on intel property and R&D wouldn’t be dumb enough to give the actual schematics to some random intern building an app. There are some serious security controls and mNDA around that data, for all internal parties, external parties and suppliers. Think about it, they must collaborate with dozens if not hundreds of parties to get to a finished products, and some amount of information must be circulating to achieve the purpose of their relationships. If those were to leak - they’d leak to the competitors first and it’d be a disaster for the entirety of the upcoming year(s) as the improvements tied to their new designs would now be considered available on all cars. It doesn’t take two dummies to think about that, soooo.. to avoid losing hundreds of millions, what do you do as an organization? Data classification, risk assessments, tight security controls, procedures, and finally mNDAs for the instances when data do have to be shared.
Furthermore, as it was pointed out on Reddit, the model appears to be made of parts that are valid for 2020 regulations, some parts for 2021.. in other words, it’s just scrap.
Title is sensationalist, while the news really ain't a big deal IHMO. Adversarial attacks against Neural Networks, or any other heuristics for that matter, are widely recognized in the industry.
Frameworks/papers aiming at benchmarking and fine tuning networks against those attacks even exist (e.g. github.com/QData/TextAttack in the case of NLP).
The tools will simply be adapted. Either to a/ identify the statistical anomalies/artifacts introduced by the manipulation process, or b/ ignore the manipulation through further training or further data normalization.
Head of DevOps at a major financial exchange where latency & resiliency is at the heart of our business, and yes, we pay Cloudflare millions. I see two things here:
# Just be ready
Most definitely not the first time Cloudflare has had trouble, just like any other system: it will fail eventually. If you're complaining about the outage, ask yourself the question: why were not you prepared for this eventuality?
Spread your name servers, and use short-TTL weighted CNAMEs, defaulting to say, 99% Cloudflare, 1% your internal load balancer. The minute Cloudflare seems problematic, make it 0% 100% to bypass Cloudflare’s infrastructure completely. This should be tested periodically to ensure that your backends are able to scale & take the load without shedding due to the lack of CDN.
# Management practices
Cloudflare's core business is networking. It actually embarrasses me to see that Cloudflare YOLO'd a BGP change in a Juniper terminal without peer reviews and/or without a proper administration dashboard, exposing safe(guarded) operations, a simulation engine and co.? In particular, re-routing traffic / bypassing POPs must be a frequent task at scale, how can that not be automated so to avoid human mistakes?
If you look at the power rails of serious data centers out there, you will quickly notice that those systems, although built 3x for the purpose of still being redundant during maintenance periods, are heavily safeguarded and automated. While technicians often have to replace power elements, the maintenance access is highly restricted with unsafe functions tiered behind physical restrictions. An example of a common function that's safeguarded is the automatic denial of an input command that would shift electrical load onto lines beyond their designed capacity - which could happen by mistake if the technician made a bad assumption (e.g. load sharing line is up while it's down) or if the assumption became violated since last check (e.g. load sharing line was up when checked, became down at a later time - milliseconds before the input even).
Q is one most important language here at BitMEX. Complicated or not, it's on KDB that all our trades happen, up to $8B in 24H volume last summer - it's highly efficient at its task, and never made us or any of our customers, loose a single Satoshi.
I have never heard of that. Could you please try again? If you can repro, open an issue and I'll be glad to fix it. Clair now tells the layer in which a vulnerability has been detected, the package name and version that cause it and how it can be fixed. It should definitely help.
Not to mention we’ve also had our fair share of production triple faults from bugs in the Intel firmware patches for Spectre, which took weeks to investigate & fix between ourselves struggling to keep our exchange up & running, Intel, and AWS.
And that is why there’s value in the CoreOS/ContainerLinux-like solutions we designed & implemented nearly a decade ago now. Being able to promptly rollback any kernel/system/package upgrades at once - either manually or either after it’s detected a few panics in quick successions is actually quite awesome. Not to mention the slow update rollout strategy baked into the Omaha controller.
But the reality is that the what-ifs are always the hardest to market, nearly always after-thoughts and with fast-spiking/fast-decaying traction after major events.