It is a useful paper, yes. Good examples, good diagnosis of issues in systems theory, good definition of a way forward.
However it suffers from (a) weak definitions and (b) implausible or strange descriptions.
For (a), what is a "strong feedback loop"? Does it have high gain (low error) or high bandwidth (fast)? Is it hidden (the accidental link imbalance example)? Is it obvious (the cold cache example)? What makes it "strong"?
Or, conversely, what is a "weak" feedback loop?
A number of acronyms are undefined (SRE, LIFO). I think I know what they mean, and most HN readers will too. What about the other readers?
And using Wikipedia to define metastability? There must be a more persistent or academically defendable reference. Wikipedia is OK for informal definitions. In a paper calling for more academic studies this is ironic.
(b) Section 2.1 "When replicas are sharded differently..." Huh?
Section 4 "upper bound" used as a verb. Should be "limit or place bounds on".
Section 4 "The strength of the loop depends on a host of constant factors from the environment..." Odd, the term is not defined but this is the second dependency listed. Very strange.
In short it needs/needed a better reviewer.
That all said, it has summarized a lot of good ideas on controlling stability in distributed systems.
Other references may be found in Adrian Colyer's "the morning paper". No longer updated but has many years of good references. See blog.acolyer.org.
Especially since Netflix has a lot of entertainment featuring black actors, producers etc. from the US. And a lot of Nigerian and South African productions too.
I'm sure that Solarwinds uses many more developer tools than what Jetbrains supplies. But only Jetbrains was "founded by Russians" so the NYT leads with that.
Question: did you unmount/mount the source filesystem between tests? When the size of data is less than RAM you are measuring cache speed, not so much disk speed. (And yes I realize that you have SSDs.)
>>[The] defenses include obvious technical components (e.g. backup systems, ‘safety’ features of equipment) and human components (e.g. training, knowledge) but also a variety of organizational, institutional, and regulatory defenses (e.g. policies and procedures, certification, work rules, team training).
This omits "design" for defenses against problems.
Example: the chemical industries in many countries in the 1960s had horrendous accident records: many employees were dying on the job. (For many reasons) the owners re-engineered their plants to substantially reduce overall accident rates. "Days since a lost-time accident" became a key performance indicator.
A key engineering process was introduced: HAZOP. The chemical flows were evaluated under all conditions: full-on, full-stop, and any other situations contrary to the design. Hazards from equipment failures or operational mistakes are thus identified and the design is adjusted to mitigate them. This was s.o.p. in the 1980s. See Wikipedia for an intro.
Similar approaches could help IT and other systems.
There is a lot marketing bullshit in the Capella articles.
For example, the Singapore image of buildings and other detections by the SAR is clearly overlaid on an optical image with trees and bushes and respective shadows. Trees and bushes are invisible to X-band (10GHz) SAR. It has to be an optical image underlaying the radar data.
Look at it. The detections by SAR are bright white. Most of the image is grey-scale showing background items.
It is not advertised as such. As an interpretable image, perhaps this is an improvement over notoriously hard-to-understand monochrome SAR imagery. BUT! It is not described as such. Hence my subjective evaluation as "bullshit".
>Is there a field of study/practice that deals with such changes to large scale system? It does sound like a very useful thing to systematically study, if possible.
"Analysis points to ways engineering strategies could be reimagined to minimize delays and other unanticipated expenses."
(from comments below).
The two quoted authors in the press release are a prof in energy studies and a nuclear engineer.
What I notice is that MIT does not have a Systems Engineering undergraduate department, but numerous specialties.
In a large design/build project such as a nuclear power plant, the systems engineering group (there must be exactly one) keeps track of the performance and function of each of the subsystems (civil works like containment, basements, buildings; electrical; controls; HVAC; the nuclear bits; and so on). In addition, it is the system engineering group that responds (or directs responses) to a change order request, and the overall impact on expense, on functionality/reliability/safety, and schedule.
It seems these are responsibilities that are not identified here with a known role; instead the authors reinvent system engineering for themselves.
There is a lot of SE work done in numerous industries (Elon Musk is what I would call a systems engineer, based on how he identifies things to do and how he gets them done. His degree is not in SE though).
A short list of US universities offer SE or related disciplines; in California/New York/Illinois. MIT is not one: it has a research group not a degree-granting program.
References:
https://www.incose.org/ The International Council of System Engineering. Has published a handbook in numerous editions over the years.
Interesting. I did some (IT) work for a company making photo plotters for the PCB industry. The MDA Fire 9000, later Cymbolic Sciences and other companies. This was about 1985.
They had a minicomputer with a hardware rasterizer driving a laser writing to photosensitive film. They had amazing throughput for the day.
One of the issues was temperature and humidity stability of the film. It would change dimensions with as little as 10% change in humidity.
The change was more than the resolution of the laser.
Also no mitigation suggested other than CSP for which "A lot of CSP policies don't..." Which is a suggestion to use CSP correctly, in a backhanded way.
I disagree. I think the purpose of 5G is to allow carriers (formerly known as The Phone Company) to sell value-added services to other companies large and small. IoT, video-on-demand, Big Data, AI, automation, etc. The bandwidth story is the distraction.
However it suffers from (a) weak definitions and (b) implausible or strange descriptions.
For (a), what is a "strong feedback loop"? Does it have high gain (low error) or high bandwidth (fast)? Is it hidden (the accidental link imbalance example)? Is it obvious (the cold cache example)? What makes it "strong"?
Or, conversely, what is a "weak" feedback loop?
A number of acronyms are undefined (SRE, LIFO). I think I know what they mean, and most HN readers will too. What about the other readers?
And using Wikipedia to define metastability? There must be a more persistent or academically defendable reference. Wikipedia is OK for informal definitions. In a paper calling for more academic studies this is ironic.
(b) Section 2.1 "When replicas are sharded differently..." Huh?
Section 4 "upper bound" used as a verb. Should be "limit or place bounds on".
Section 4 "The strength of the loop depends on a host of constant factors from the environment..." Odd, the term is not defined but this is the second dependency listed. Very strange.
In short it needs/needed a better reviewer.
That all said, it has summarized a lot of good ideas on controlling stability in distributed systems.
Other references may be found in Adrian Colyer's "the morning paper". No longer updated but has many years of good references. See blog.acolyer.org.