Software updates: the “unknown unknown” of the replication crisis(blogs.lse.ac.uk)
blogs.lse.ac.uk
Software updates: the “unknown unknown” of the replication crisis
http://blogs.lse.ac.uk/impactofsocialsciences/2018/06/07/software-updates-the-unknown-unknown-of-the-replication-crisis/
28 comments
I'll concede that the story is different when you're relying on commercial software that you've purchased to implement an algorithm correctly, but in the limited case of replicating results (whether they're flawed or not), the onus really should be on the user to record what version of software they're using and how they configured it. The article does a lot of gymnastics to make software correctness and research reproducibility feel like the same issue.
> the onus really should be on the user to record what version of software they're using and how they configured it.
Recording version, etc. is fine, but what use is it if it's not actually possible to install that software to reproduce whatever results were obtained? (This also extends to hardware, btw., but VMs can probably attain enough fidelity for that to not be an issue as long as the physical media are still readable.)
Example: Reasearch done on Windows XP at some point in time past SP3. Unless you're prepared well ahead of EoL on Windows XP there's really no way to reproduce that environment.
(I'll definitely grant that it's probably unlikely that the obscure details of the OS have that much impact on whatever results were obtained by programs running atop it, but it's at least moderately plausible.)
Recording version, etc. is fine, but what use is it if it's not actually possible to install that software to reproduce whatever results were obtained? (This also extends to hardware, btw., but VMs can probably attain enough fidelity for that to not be an issue as long as the physical media are still readable.)
Example: Reasearch done on Windows XP at some point in time past SP3. Unless you're prepared well ahead of EoL on Windows XP there's really no way to reproduce that environment.
(I'll definitely grant that it's probably unlikely that the obscure details of the OS have that much impact on whatever results were obtained by programs running atop it, but it's at least moderately plausible.)
Versioning/Configuration in industry that has automated some aspect of their build process to include external dependencies leads to label accumulation (I'm intuiting that this is greater than exponential growth - assuming there exist dependencies for a system A that are dependent on other dependencies that system A is dependent on).
The problem is you can't document everything while also trying to automate everything. It's always a balance of figuring out which information is most important, and if 'oneself' were to attempt to construct a model that represents everything, 'oneself' would simply be stuck in a catatonic state of analysis paralysis, in perpetuity and invariably, until the heat death of the universe.
Yes, the onus is on the user, but who is the 'user' in the industry of development? How do you maintain structure in records for information systems that are fundamentally, at core, completely opened%? I believe this problem maps back to the science of collecting and verifying the veracity of information, up to isomorphism (quite literally, this is the definition of the graph isomorphism problem).
% The intrinsic nature of science and discovery is 'we discover new things because we assume there is more to know (~ we don't know everything)'. Therefore, all components of such a system are subject to change. There is nothing that can be said to be permanent in knowledge discovery systems where the premise of discovery rests on 'assertions' that can not be provably demonstrated to be factual AND absolute.
The problem is you can't document everything while also trying to automate everything. It's always a balance of figuring out which information is most important, and if 'oneself' were to attempt to construct a model that represents everything, 'oneself' would simply be stuck in a catatonic state of analysis paralysis, in perpetuity and invariably, until the heat death of the universe.
Yes, the onus is on the user, but who is the 'user' in the industry of development? How do you maintain structure in records for information systems that are fundamentally, at core, completely opened%? I believe this problem maps back to the science of collecting and verifying the veracity of information, up to isomorphism (quite literally, this is the definition of the graph isomorphism problem).
% The intrinsic nature of science and discovery is 'we discover new things because we assume there is more to know (~ we don't know everything)'. Therefore, all components of such a system are subject to change. There is nothing that can be said to be permanent in knowledge discovery systems where the premise of discovery rests on 'assertions' that can not be provably demonstrated to be factual AND absolute.
"Versioning/Configuration in industry that has automated some aspect of their build process to include external dependencies leads to label accumulation"
Okay, but most software shops shove all those labels into their product version, usually something simple like "ToolFoo 3.4.11".
I think that this numbering and basic "here's the binary, run at your own risk and don't call me" legacy support are enough on the vendor's part. And I agree with gp that if the researcher fails to document the top-level tool version they're using, plus the changes they made to default configuration, then it's the researcher's fault.
Don't you agree?
Okay, but most software shops shove all those labels into their product version, usually something simple like "ToolFoo 3.4.11".
I think that this numbering and basic "here's the binary, run at your own risk and don't call me" legacy support are enough on the vendor's part. And I agree with gp that if the researcher fails to document the top-level tool version they're using, plus the changes they made to default configuration, then it's the researcher's fault.
Don't you agree?
There are many software stacks that are not versioned only at the top level, especially in scientific software.
As an example: If I tell you I calculated a result on SciPy 1.1.0, do you know what version of what code is doing the linear algebra? No, because there's no single answer. It depends how SciPy was built and what's installed on the rest of my system. It could be OpenBLAS, ATLAS, MKL...
So if there were a bug in, say, a particular version of OpenBLAS, and it manifests itself in SciPy, there's no way to tell whether the bug is present based on the SciPy version.
As an example: If I tell you I calculated a result on SciPy 1.1.0, do you know what version of what code is doing the linear algebra? No, because there's no single answer. It depends how SciPy was built and what's installed on the rest of my system. It could be OpenBLAS, ATLAS, MKL...
So if there were a bug in, say, a particular version of OpenBLAS, and it manifests itself in SciPy, there's no way to tell whether the bug is present based on the SciPy version.
I don't know if I can agree with that.
Pretend that all of that has been automated - you have some dependency management system that auto-documents the versions of all software you are using.
How do you check that all of that documentation is correct across all systems with one singular 'system'? There's the obvious - make everyone perfect and know everything they need to know for the work they do. The individual is responsible. Okay, but how do you check that all of that is functioning the way it's supposed to? It literally simply creates a copy of the original problem.
There's the idea to put the onus on the individual researcher or developer for managing their specific 'part', but the problem is that their specific part is coupled with other parts they do not have a complete understanding of, and may never have a complete understanding of. It's literally uncertain whether it's possible to actually have a complete understanding.
Oneself can tell oneself all they want "I know all of the things I need to know" but I think when oneself does that, oneself forgets how to know they might not know all of the things they need to know. But that might just be the insanity inside of me speaking. I know I don't know everything, even the things I think I'm supposed to know, I don't know if I know them. Because as a developer working with other developers, code changes, interpretation of code changes, there's always variation in understanding 'the model' everyone is working off of and working towards. This seems to influence things, and what if all that does is converge understanding towards self proving (e.g. reconstructing a duplicate of the original data) but it's done in a way where the simple mere fact that everyone has agreed to hold those 'models/beliefs' as strongest is what causes those beliefs to manifest externally?
Is that actually provably correct? I don't think so. I just think people forget how to know the difference between what is a 'bug/feature' of the nature of knowing and thinking, and what is an error.
This may be tangential to the nature of software correctness and code, but 1. we don't know how programs get used in the future unless the person doing the programming is also the only person using the program. and 2. This bug/feature nature of knowing and thinking is what allows research in things like the social sciences% to not finalize permanently into some static model that is unchanging til the end of time. You can balance both of these awarenesses but, it's not a trivial understanding. It's very, very complicated and it makes the most mundane details of life seem like sheer insanity.
% referring to
> The replication crisis that continues to engulf the social sciences% is largely concerned with the known problems,
Pretend that all of that has been automated - you have some dependency management system that auto-documents the versions of all software you are using.
How do you check that all of that documentation is correct across all systems with one singular 'system'? There's the obvious - make everyone perfect and know everything they need to know for the work they do. The individual is responsible. Okay, but how do you check that all of that is functioning the way it's supposed to? It literally simply creates a copy of the original problem.
There's the idea to put the onus on the individual researcher or developer for managing their specific 'part', but the problem is that their specific part is coupled with other parts they do not have a complete understanding of, and may never have a complete understanding of. It's literally uncertain whether it's possible to actually have a complete understanding.
Oneself can tell oneself all they want "I know all of the things I need to know" but I think when oneself does that, oneself forgets how to know they might not know all of the things they need to know. But that might just be the insanity inside of me speaking. I know I don't know everything, even the things I think I'm supposed to know, I don't know if I know them. Because as a developer working with other developers, code changes, interpretation of code changes, there's always variation in understanding 'the model' everyone is working off of and working towards. This seems to influence things, and what if all that does is converge understanding towards self proving (e.g. reconstructing a duplicate of the original data) but it's done in a way where the simple mere fact that everyone has agreed to hold those 'models/beliefs' as strongest is what causes those beliefs to manifest externally?
Is that actually provably correct? I don't think so. I just think people forget how to know the difference between what is a 'bug/feature' of the nature of knowing and thinking, and what is an error.
This may be tangential to the nature of software correctness and code, but 1. we don't know how programs get used in the future unless the person doing the programming is also the only person using the program. and 2. This bug/feature nature of knowing and thinking is what allows research in things like the social sciences% to not finalize permanently into some static model that is unchanging til the end of time. You can balance both of these awarenesses but, it's not a trivial understanding. It's very, very complicated and it makes the most mundane details of life seem like sheer insanity.
% referring to
> The replication crisis that continues to engulf the social sciences% is largely concerned with the known problems,
People are latching onto different individual takeaways from this, but it really requires a multi-prong approach (and can never be fixed completely):
Companies making scientific software should test at a higher level of rigor than others. There's no such thing as bug-free software, but that on missile launch systems and NASA satellites comes pretty close, so scientific software can too.
For algorithms that go beyond just math, researchers should consider writing their own code.
As applicable, software version numbers (and perhaps binary hashes) should be listed as part of the reproduction conditions in papers.
Companies making scientific software should test at a higher level of rigor than others. There's no such thing as bug-free software, but that on missile launch systems and NASA satellites comes pretty close, so scientific software can too.
For algorithms that go beyond just math, researchers should consider writing their own code.
As applicable, software version numbers (and perhaps binary hashes) should be listed as part of the reproduction conditions in papers.
> Furthermore, perhaps before releasing a new version of the software for a broader usage, these companies should ensure it is bug-free by pre-testing it and thus guaranteeing the correctness of the produced estimations.
This is f-king ridiculous. This will never happen. There will always be bugs. Bug free.
If you don't want bugs, write the code yourself, or read all of the code yourself. After you do that, try to figure out whether your mind can still parse information 'correctly' relative to your respective science.
This is f-king ridiculous. This will never happen. There will always be bugs. Bug free.
If you don't want bugs, write the code yourself, or read all of the code yourself. After you do that, try to figure out whether your mind can still parse information 'correctly' relative to your respective science.
Suggesting bug-free software demonstrates an ignorance of how software engineering works, but requesting more rigorous testing not just for absolute correctness, but perhaps even for consistency with past versions (a novel concept) is perfectly reasonable.
> not just for absolute correctness, but perhaps even for consistency with past versions (a novel concept)
Sorry, can you explain what you mean by this further?
Sorry, can you explain what you mean by this further?
Normally, software functions are only tested against what is considered to be the universally correct outcome. The wiggle room - i.e., undefined aspects of the behavior - within that definition of "correct" is not tested. It would be interesting for testing of scientific software to incorporate a second metric: "how close is it to what it was last time (assuming last time was also 'correct')?" For the sake of reproducibility, maybe it's meaningful whether or not the current version of correctness is in line with the previous version of correctness.
There's virtually no other domain in which you'd want to test software this way, which is what makes it novel and interesting.
For example, maybe a floating point operation rounds the very last digit down. Then through code changes, it starts truncating the very last digit instead. Your tests may still consider this result "correct", but it might be worth considering the fact that it differed from the previous "correct" result.
There's virtually no other domain in which you'd want to test software this way, which is what makes it novel and interesting.
For example, maybe a floating point operation rounds the very last digit down. Then through code changes, it starts truncating the very last digit instead. Your tests may still consider this result "correct", but it might be worth considering the fact that it differed from the previous "correct" result.
Or, maybe your "equals" function, when given two floats (floats are typically considered not-comparable in most languages), happens to return true in a particular case, then later on through code changes, it ends up returning false for the same two numbers. Since <float> == <float> is undefined, neither behavior is "incorrect", but it might be worthwhile to at least keep the behavior consistent.
I have the impression that issues with a software version not matching a previous version are not only not novel, but have been plentiful with any kind of software since the beginning of time. Microsoft is proverbial for Windows being under the constraint of maintaining past behavior however imperfect.
To wit, it's such a common thing that there is an xkcd for it:
https://imgs.xkcd.com/comics/workflow.png
P.S. I was momentarily repressing my first hand experience, but in a past job, there were many times I wanted to fix things and wasn't allowed to, because if the results changed, it would upset someone. The cardinal, CYA rule, was if we've been reporting something wrong for years, it better be consistent forever. Which is really painful if you care about correctness and compulsively investigate it.
To wit, it's such a common thing that there is an xkcd for it:
https://imgs.xkcd.com/comics/workflow.png
P.S. I was momentarily repressing my first hand experience, but in a past job, there were many times I wanted to fix things and wasn't allowed to, because if the results changed, it would upset someone. The cardinal, CYA rule, was if we've been reporting something wrong for years, it better be consistent forever. Which is really painful if you care about correctness and compulsively investigate it.
That sounds awful.
Specifically for social sciences, wouldn't this mechanic be able to be exploited in the same way p-values are?
In the case of software testing, I think I agree with Dijkstra on this one, except for different reasons:
> We could, for instance, begin with cleaning up our language by no longer calling a bug a bug but by calling it an error. It is much more honest because it squarely puts the blame where it belongs, viz. with the programmer who made the error. The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking is intellectually dishonest as it disguises that the error is the programmer's own creation. The nice thing of this simple change of vocabulary is that it has such a profound effect: while, before, a program with only one bug used to be "almost correct", afterwards a program with an error is just "wrong" (because in error).
I don't think 'blame' is the right word, or calling the 'programmer' who made the error intellectually dishonest. A fuzzy 'language' might be helpful for 'developers' who are not quite developers. A developer would categorize such an occurrence in discrete terminology - WARNING, ERROR, INFO, DEBUG, coupled with checks and tests throughout the code - in order to retain discrete structure across code, which is much easier to reason about mathematically (and much easier to isolate the source of an error), when such discrete objects and discrete structures stack.
The problem is you still need these structures in the code. You still need to include the tests in the code. You still need to know what to look for, in the code. You still need to know where the code might change, before you know where the code has changed. I don't think it's a futile effort, but I suppose I'm scratching my head a little on how this solves the problem? It's much simpler to deal with a compilation error, or any kind of explicit error that prevents some aspect of the program from running in a way that is defined as invalid. It's annoying, but it at least says "definitely broken code", instead of "maybe broken code?" which may run for a very long time, accumulating "maybe errors?". IMHO, that behavior can be perverted quite easily, because, some "maybe errors" can be ignored, other "maybe errors" can be focused on, depending on who is looking, and who isn't.
I honestly think if social scientists are going to complain about code, they need to learn how to code. A computer isn't a magic fantasy box that just confirms beliefs. It takes effort, skill, hard work, and honestly, dealing with bugs you can't always determine the origin of is very humbling (you, other developer?), so they should learn some of that too. And I'm sure they'd say "we already do!" but there we are again, going full circle, choosing to believe what we can't prove, but tricking ourselves into finding a magical way to prove what we wanted to prove, about ourselves, about others, etc. Bugs will happen. It's better when they are clearly defined, because at least the source of the error can be isolated, tested, identified, and corrected. Stacks on stacks of 'maybe error?' can lead to it literally conveying information that is the opposite of what it intends to.
There needs to be some compassion for people who are developers. We can't know where all the bugs are all the time. The best we can do is make code as bug-free as we possibly can. Coding is just hard! You think you get better at it over time, but it's really easy to be chugging along, coding coding coding, and all of a sudden - bug you didn't think of! And that just, it's humbling - and you know - this is your life, this is the path you chose. This is what you will be doing until you stop coding. Social scientists need to be nicer to developers. Same problems they deal with, just shifted.
> Testing shows the presence, not the absence of bugs (Dijkstra)
In the case of software testing, I think I agree with Dijkstra on this one, except for different reasons:
> We could, for instance, begin with cleaning up our language by no longer calling a bug a bug but by calling it an error. It is much more honest because it squarely puts the blame where it belongs, viz. with the programmer who made the error. The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking is intellectually dishonest as it disguises that the error is the programmer's own creation. The nice thing of this simple change of vocabulary is that it has such a profound effect: while, before, a program with only one bug used to be "almost correct", afterwards a program with an error is just "wrong" (because in error).
I don't think 'blame' is the right word, or calling the 'programmer' who made the error intellectually dishonest. A fuzzy 'language' might be helpful for 'developers' who are not quite developers. A developer would categorize such an occurrence in discrete terminology - WARNING, ERROR, INFO, DEBUG, coupled with checks and tests throughout the code - in order to retain discrete structure across code, which is much easier to reason about mathematically (and much easier to isolate the source of an error), when such discrete objects and discrete structures stack.
The problem is you still need these structures in the code. You still need to include the tests in the code. You still need to know what to look for, in the code. You still need to know where the code might change, before you know where the code has changed. I don't think it's a futile effort, but I suppose I'm scratching my head a little on how this solves the problem? It's much simpler to deal with a compilation error, or any kind of explicit error that prevents some aspect of the program from running in a way that is defined as invalid. It's annoying, but it at least says "definitely broken code", instead of "maybe broken code?" which may run for a very long time, accumulating "maybe errors?". IMHO, that behavior can be perverted quite easily, because, some "maybe errors" can be ignored, other "maybe errors" can be focused on, depending on who is looking, and who isn't.
I honestly think if social scientists are going to complain about code, they need to learn how to code. A computer isn't a magic fantasy box that just confirms beliefs. It takes effort, skill, hard work, and honestly, dealing with bugs you can't always determine the origin of is very humbling (you, other developer?), so they should learn some of that too. And I'm sure they'd say "we already do!" but there we are again, going full circle, choosing to believe what we can't prove, but tricking ourselves into finding a magical way to prove what we wanted to prove, about ourselves, about others, etc. Bugs will happen. It's better when they are clearly defined, because at least the source of the error can be isolated, tested, identified, and corrected. Stacks on stacks of 'maybe error?' can lead to it literally conveying information that is the opposite of what it intends to.
There needs to be some compassion for people who are developers. We can't know where all the bugs are all the time. The best we can do is make code as bug-free as we possibly can. Coding is just hard! You think you get better at it over time, but it's really easy to be chugging along, coding coding coding, and all of a sudden - bug you didn't think of! And that just, it's humbling - and you know - this is your life, this is the path you chose. This is what you will be doing until you stop coding. Social scientists need to be nicer to developers. Same problems they deal with, just shifted.
> Testing shows the presence, not the absence of bugs (Dijkstra)
I disagree with calling bugs "errors" that the programmer made. A program has a context, and bugs depend on the context, hence the old joke about "features". For a really simple example, suppose I write a program to do mail merge for myself and it has my name hard-coded in it. If you try to use it, now it has a bug, because it prints the wrong name. But it doesn't have a bug when I use it.
And if the context determines whether a bug exists, one can deduce that even if a program is provably correct by some standard and works for a billion years, the context could then change and oops, it has a bug.
And if the context determines whether a bug exists, one can deduce that even if a program is provably correct by some standard and works for a billion years, the context could then change and oops, it has a bug.
It's called an error when it's being used in a way it wasn't intended to be used. That matters when future you forgets how you programmed your code.
If the statistical significance of your results is algorithm-dependent, shouldn't they be regarded as suspect? Perhaps it might be just a failure of imagination on my part, but I find it odd to think that changing a software package might budge estimates far enough to push them outside the zone of statistical significance unless they were only marginally significant in the first place.
I could imagine algorithmic differences adding bias in error margins. Both versions might be accurate approximations of the answer, but one might lean towards one end of the error space and one might lean towards the other.
It's like when fixing a bug in library code breaks application code. Usually it's because there was some undefined behavior in the library - which wasn't part of the contract - which the application (knowingly or unknowingly) relied upon, and then the updated version produces a different undefined behavior.
It's like when fixing a bug in library code breaks application code. Usually it's because there was some undefined behavior in the library - which wasn't part of the contract - which the application (knowingly or unknowingly) relied upon, and then the updated version produces a different undefined behavior.
> If the statistical significance of your results is algorithm-dependent, shouldn't they be regarded as suspect?
It's a fair question.
Here's a possible recipe to get such variations in estimates: (1) an estimator that does not really match the distribution of the dependent variable; (2) small sample sizes with insufficiently well-handled influential observations; (3) (robust) standard error corrections leading to disproportionate confidence intervals; and (4) limited work on diagnostics, which is another way to make all of the previous points.
The points above can be used to 'take down' many papers published in journals like the one in which the incident happened, but you can also take a more charitable view and rescue most of those papers by claiming, for instance, that statistical significance should not govern (and even less govern alone) over the identification of the data generation process.
My conclusion is therefore: yes, algorithmic variation makes those results suspect, but on a single dimension that should probably not stand as the most important one in assessing those results in the first place.
Edited: syntax.
It's a fair question.
Here's a possible recipe to get such variations in estimates: (1) an estimator that does not really match the distribution of the dependent variable; (2) small sample sizes with insufficiently well-handled influential observations; (3) (robust) standard error corrections leading to disproportionate confidence intervals; and (4) limited work on diagnostics, which is another way to make all of the previous points.
The points above can be used to 'take down' many papers published in journals like the one in which the incident happened, but you can also take a more charitable view and rescue most of those papers by claiming, for instance, that statistical significance should not govern (and even less govern alone) over the identification of the data generation process.
My conclusion is therefore: yes, algorithmic variation makes those results suspect, but on a single dimension that should probably not stand as the most important one in assessing those results in the first place.
Edited: syntax.
> Yet, the company does not justify which version of the program is the correct one to use in order to get as close as possible to the underlying true relationship.
Even if they did that's still not good enough. I guess it's my own ignorance of how science is done, but I'd expect a higher standard of certainty for which algorithm you're using than "some vendor said so".
Even if they did that's still not good enough. I guess it's my own ignorance of how science is done, but I'd expect a higher standard of certainty for which algorithm you're using than "some vendor said so".
One project I have worked with was analyzing data in the education industry. Mainly public schools, think student grades and educational performance in general.
You'd think a simple percentage grade would have meaning, but it doesn't. Often times federal and state rules alter how data is to be "perceived" and therefore, analayzed and reported on.
There are a few major players in the data collection and analysis game for public school system, and they all not only have their own way of determining skill level (eg, reading skill level) with their own testing systems, they also analyze their custom data with their own algorithms. Thus leading to quite a variation in their ultimate recommendations/conclusions between vendors.
Then, take into account some schools have many students with English as a second language, and based on how the state views that school, it may be analyzed as "all equal" or take into account the language barriers.
Then mix in that you have to analyze students over multiple years, with students entering and leaving districts, sometimes with data vendors changing, etc... Then there is state and federal laws mixing with multiple vendors data collection _and_ analysis.
So the next time you hear a particular school district is under/over performing based on these vendor's analysis (especially related to budgets), keep in mind how variant the actual reality is to what is being reported.
Maybe every complex system has the same core problems?
You'd think a simple percentage grade would have meaning, but it doesn't. Often times federal and state rules alter how data is to be "perceived" and therefore, analayzed and reported on.
There are a few major players in the data collection and analysis game for public school system, and they all not only have their own way of determining skill level (eg, reading skill level) with their own testing systems, they also analyze their custom data with their own algorithms. Thus leading to quite a variation in their ultimate recommendations/conclusions between vendors.
Then, take into account some schools have many students with English as a second language, and based on how the state views that school, it may be analyzed as "all equal" or take into account the language barriers.
Then mix in that you have to analyze students over multiple years, with students entering and leaving districts, sometimes with data vendors changing, etc... Then there is state and federal laws mixing with multiple vendors data collection _and_ analysis.
So the next time you hear a particular school district is under/over performing based on these vendor's analysis (especially related to budgets), keep in mind how variant the actual reality is to what is being reported.
Maybe every complex system has the same core problems?
>"The replication crisis is largely concerned with known problems, such as the lack of replication standards, non-availability of data, or p-hacking. One hitherto unknown problem is the potential for software companies’ changes to the algorithms used for calculations to cause discrepancies between two sets of reported results."
- June 7th, 2018
They didnt know about this?
They didnt know about this?
No, they expect developers to know everything, apparently. At least to know everything that doesn't give them a dopamine hit when two result sets match, validating their theories/beliefs.
This is what developers deal with all the time, so this is obvious. Code should work this way, but it doesn't always. Yes, we can all make heroes of ourselves by thinking of ourselves as saviors for those edge cases of catastrophic failure, but we don't, because we are aware, that bugs happen - they are a mathematical certainty assuming one doesn't have access to all information, including every single use case of a program in the future.
This is what developers deal with all the time, so this is obvious. Code should work this way, but it doesn't always. Yes, we can all make heroes of ourselves by thinking of ourselves as saviors for those edge cases of catastrophic failure, but we don't, because we are aware, that bugs happen - they are a mathematical certainty assuming one doesn't have access to all information, including every single use case of a program in the future.
I mean almost the first thing that happened to me when I tried to use commercial stats software was being unable to reproduce a calculation manually. Eventually found out they had a known bug in their code that was left in for ~20 years and when they fixed it they called it the "enhanced version". The calculation was based on the original version of a paper that was retracted and corrected. I would never trust commercial stats software after that.
I really mean this was close to the very first thing, so I would assume it has to be very common.
I really mean this was close to the very first thing, so I would assume it has to be very common.
It seems to me a lot could be solved by making data available and compiling software to webasm, then packaging up a single web page that demonstrates any algorithms used.
Maybe they should also be required to publish their software environment as a container or other virtual package.
> Speaking in 2002 about weapons of mass destruction, United States Secretary of Defense Donald Rumsfeld infamously distinguished between the “known unknowns” and the “unknown unknowns”
Rumsfeld rather famously distinguished, as the world has proved by endlessly quoting and paraphrasing him.
The infamous appellation here is presumably based solely on the authors' dislike of Rumsfeld. I stopped reading.
Rumsfeld rather famously distinguished, as the world has proved by endlessly quoting and paraphrasing him.
The infamous appellation here is presumably based solely on the authors' dislike of Rumsfeld. I stopped reading.