Linux Foundation’s demands to the University of Minnesota(zdnet.com)
zdnet.com
Linux Foundation’s demands to the University of Minnesota
https://www.zdnet.com/article/the-linux-foundations-demands-to-the-university-of-minnesota-for-its-bad-linux-patches/
178 comments
ABET accredited schools already require "ethics". Most of these classes are: Emmanuel Kant existed and "don't copy that floppy." I'm sure there are some very competent professors from the philosophy departments whom do a very competent job, but I'm speaking in generalities.
The fact is that if someone's employer, like an academic advisor, asks you to do something "unethical" most of that training is moot.
The fact is that if someone's employer, like an academic advisor, asks you to do something "unethical" most of that training is moot.
The ethics coursework at my ABET accredited alma mater, which was a 1 credit joke BTW, mostly covered plagiarism.
Human ethics are basically not covered at all. There is a huge ideological blindspot inside engineering departments - if my work is thorough and scientifically sound that's enough. The process is elevated above the final result, especially the final product's interaction with the wider world.
You can build a nuclear bomb following that mindset, but people just seem to miss it.
Human ethics are basically not covered at all. There is a huge ideological blindspot inside engineering departments - if my work is thorough and scientifically sound that's enough. The process is elevated above the final result, especially the final product's interaction with the wider world.
You can build a nuclear bomb following that mindset, but people just seem to miss it.
I sat on the curriculum committee for my university's (one of the Big 10) CS program for a year as an undergrad, and got to see first-hand some of the justifications from the faculty for how certain courses met the requirements of the ABET program.
It was... often something of a stretch, especially when it came to "soft skills" like ethics and whatnot. As I remember, the "justification" for meeting the ABET ethics requirements was spread across at least three courses, the first of which was a 1 credit lecture course that everyone in the major (including myself) blew off.
To be fair, we were already having difficulty cramming all the ABET requirements into the course material, to say nothing of related software development skills and tools. Our program was struggling to accommodate the insane growth we were already seeing in students--enrollments grew by something like 70% over six years, and we were struggling to hire sufficient faculty. When I graduated we were leaning heavily on online-only introductory courses and grading automation, which has its own issues.
It was... often something of a stretch, especially when it came to "soft skills" like ethics and whatnot. As I remember, the "justification" for meeting the ABET ethics requirements was spread across at least three courses, the first of which was a 1 credit lecture course that everyone in the major (including myself) blew off.
To be fair, we were already having difficulty cramming all the ABET requirements into the course material, to say nothing of related software development skills and tools. Our program was struggling to accommodate the insane growth we were already seeing in students--enrollments grew by something like 70% over six years, and we were struggling to hire sufficient faculty. When I graduated we were leaning heavily on online-only introductory courses and grading automation, which has its own issues.
Real engineers have actual professional standards bodies they're accountable too for ethical violations. Your boss telling you to do something unethical runs right up against the risk you won't be able to practice your profession. This is why the built environment largely endures.
It's only a very small minority of engineers who do. Unless you're a PE and you get to sign off on large development projects, there's no real accountability.
No matter how much teeth a governing body has, it's always going to be gun shy about denying people their livelihood. They're only going to go after the most egregious and flagrant problems. As long as the transgressions are mild, they pose no real danger to those who want to violate whatever ethics code are in place.
No matter how much teeth a governing body has, it's always going to be gun shy about denying people their livelihood. They're only going to go after the most egregious and flagrant problems. As long as the transgressions are mild, they pose no real danger to those who want to violate whatever ethics code are in place.
Why does anyone think you can get people to be ethical just by teaching them ethics? Knowing what other people say about ethics doesn't have a clear path to strengthening your conscience, which is the only "motivation" most people have to be ethical in the first place.
> Why does anyone think you can get people to be ethical just by teaching them ethics?
Teaching ethics isn’t about getting people to be ethical who aren’t inclined that way, its to make people who are basically well-intentioned aware of ethical issues in the field and get them in the habit of thinking about ethics in the context of the work, both in planning and executing their own and looking at other work in the field of interest.
> Knowing what other people say about ethics doesn't have a clear path to strengthening your conscience, which is the only "motivation" most people have to be ethical in the first place.
Teaching ethics isn’t about motivating ethical behavior any more than teaching compilers is about motivating people to write them.
Teaching ethics isn’t about getting people to be ethical who aren’t inclined that way, its to make people who are basically well-intentioned aware of ethical issues in the field and get them in the habit of thinking about ethics in the context of the work, both in planning and executing their own and looking at other work in the field of interest.
> Knowing what other people say about ethics doesn't have a clear path to strengthening your conscience, which is the only "motivation" most people have to be ethical in the first place.
Teaching ethics isn’t about motivating ethical behavior any more than teaching compilers is about motivating people to write them.
I’m going to be generous and propose that most people behave ethically most of the time, including the UMN researchers. There is a lot of discussion in this thread about the basic question of whether or not the project was ethical. Anyone can have a blind spot when making such judgements. An ethics course could help people better understand the principles of ethical behavior, which would lead to a logical shared conclusion about this.
In other words, I think it’s as much about knowledge as it is about conscience.
In other words, I think it’s as much about knowledge as it is about conscience.
The decision on the ethics of this experiment isn't up to us, nor was it up to the researchers. It should have been left to an IRB for an approval, which most likely wouldn't have approved of this research in its current state.
The lack of oversight is the problem. Because now it calls into question the work of the grad students, the papers sponsor who oversaw the work, the department, and the school as a whole. It is a significant damage to their reputation.
At some of the research places, doing unapproved research is grounds for immediate termination, regardless of tenure. As for the students, they would have been investigated, most likely expelled for ignoring protocol or denied their credits for the semesters the research took placed and put on academic probation. The department as a whole would also be looked into.
The lack of oversight is the problem. Because now it calls into question the work of the grad students, the papers sponsor who oversaw the work, the department, and the school as a whole. It is a significant damage to their reputation.
At some of the research places, doing unapproved research is grounds for immediate termination, regardless of tenure. As for the students, they would have been investigated, most likely expelled for ignoring protocol or denied their credits for the semesters the research took placed and put on academic probation. The department as a whole would also be looked into.
At least according to the researcher’s letter[1], the UMN IRB did look at the study after the fact and determined approval wasn’t needed.
> The IRB of UMN reviewed the study and determined that this is not human research (a formal IRB exempt letter was obtained).
[1] https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
> The IRB of UMN reviewed the study and determined that this is not human research (a formal IRB exempt letter was obtained).
[1] https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
> Why does anyone think you can get people to be ethical just by teaching them ethics?
Because studies say so? https://mitsloan.mit.edu/ideas-made-to-matter/can-ethics-be-...
"The central finding is striking: those passing the older exam, with more rules and ethics coverage, were 25% less likely to commit any kind of misconduct"
Because studies say so? https://mitsloan.mit.edu/ideas-made-to-matter/can-ethics-be-...
"The central finding is striking: those passing the older exam, with more rules and ethics coverage, were 25% less likely to commit any kind of misconduct"
Right: If everyone knows something is wrong, people are less likely to take the risk of getting caught. Even if it's not those bad actors who change, the bad actors don't act badly because everyone else would recognize it as wrong.
> Why does anyone think you can get people to be ethical just by teaching them ethics?
Because you can teach ethics in much the same way you teach security best practices.
Lots of developers practice security theater without actually believing it's all necessary. The same can be done with ethics.
Because you can teach ethics in much the same way you teach security best practices.
Lots of developers practice security theater without actually believing it's all necessary. The same can be done with ethics.
People who write insecure software typically didn't intend to, so you can teach them. People who do unethical things intended to do unethical things, so teaching is useless.
I would argue that Hanlon's razor applies. Most people aren't thinking about ethics, they're just trying to solve a problem.
When a product manager asks them to implement a dark pattern to trick users into clicking something, they aren't thinking "heh... fools, I'll show them", they're thinking "Sure thing boss, whatever you say". It's not that these people are unethical, it's that it's never a consideration.
I was in a meeting once where business was discussing abandonment and it was suggested that if a user filled out a form 100% but never clicked submit, it was reasonable to assume that after 20 minutes if the window was still open then we should just automatically submit the form because that was their intent.
There was no malice in the thought or question of ethics, the person who suggested it thought it was a helpful suggestion and would improve the product and reduce abandonment. When I framed it from the perspective of the end user they quickly shot down their own idea.
When a product manager asks them to implement a dark pattern to trick users into clicking something, they aren't thinking "heh... fools, I'll show them", they're thinking "Sure thing boss, whatever you say". It's not that these people are unethical, it's that it's never a consideration.
I was in a meeting once where business was discussing abandonment and it was suggested that if a user filled out a form 100% but never clicked submit, it was reasonable to assume that after 20 minutes if the window was still open then we should just automatically submit the form because that was their intent.
There was no malice in the thought or question of ethics, the person who suggested it thought it was a helpful suggestion and would improve the product and reduce abandonment. When I framed it from the perspective of the end user they quickly shot down their own idea.
You're thinking of morality. Ethics are just the rules that an organization follows. It's more about process, expectations and standards than it is about any particular moral imperative.
It might not make the companies/professors ethical, but might help the employees, grad students, or more importantly test subject notice bad behavior.
By showing what is right and wrong to those who are conscientious enough to act as watchdogs and whistleblowers.
You can't make a bad actor good by showing the rules, but you can make a neutral actor aware of exactly what needs to be reported, and give them a forum.
Without training, right and wrong are not normalized and standardized, leading to fuzzy feelings and confusion about what can be done.
You can't make a bad actor good by showing the rules, but you can make a neutral actor aware of exactly what needs to be reported, and give them a forum.
Without training, right and wrong are not normalized and standardized, leading to fuzzy feelings and confusion about what can be done.
I was on a road trip with 3 engineers this weekend and we were discussing this. I mentioned that I, along with nearly every grad student in life sciences is required to take a research ethics course because NIH training grants mandate that any student funded on the grant take an ethics course.
They all said that they wish that their graduate programs had required them to take an ethics course. One of them (from India) also noted that basic concepts such as plagiarism are assumed to be understood at the graduate level in the US, but that few students coming from Asia have received any orientation to ethical academic practice for students, much less researchers.
I don't know enough about how CS and engineering graduate programs are funded, but it seems that NSF is not funding very many programs, given that they do seem to require such training https://www.nsf.gov/bfa/dias/policy/rcr.jsp.
They all said that they wish that their graduate programs had required them to take an ethics course. One of them (from India) also noted that basic concepts such as plagiarism are assumed to be understood at the graduate level in the US, but that few students coming from Asia have received any orientation to ethical academic practice for students, much less researchers.
I don't know enough about how CS and engineering graduate programs are funded, but it seems that NSF is not funding very many programs, given that they do seem to require such training https://www.nsf.gov/bfa/dias/policy/rcr.jsp.
I'm an epidemiologist who works with a lot of computer scientists. Step #1 for collaborations is usually getting them human subjects training and familiarizing them with the IRB.
The school I attended had a half-semester ethics course that was required for all engineering PhD students. Not sure if this a common thing, though.
I agree. I've also noticed that some developers seem to have a strong sense of "we should not do that" while others don't and I'm not sure why.
One recent example was impersonating/spoofing a manager's email to send phishing emails as part of a security compliance test. One developer saw no harm at all in doing so while another felt we should ask permission first.
It's a basic violation of trust to do this sort of thing and in the very least, permission should be obtained first. Otherwise, the violation of trust and repercussions may be severe.
One recent example was impersonating/spoofing a manager's email to send phishing emails as part of a security compliance test. One developer saw no harm at all in doing so while another felt we should ask permission first.
It's a basic violation of trust to do this sort of thing and in the very least, permission should be obtained first. Otherwise, the violation of trust and repercussions may be severe.
As someone who naturally gets feelings of aversion when I hear some solutions suggested and who works with folks who can suggest them without blinking, I honestly think "should" never enters their mind. It seems to be a pure engineering-in-a-vacuum mindset where "can" is all that matters. They envision an admittedly potentially interesting end outcome and the means to get there is just engineering. They don't think twice about how their solution just involved enabling tracking everyone's phone's location to enable it.
My and many others first thought would likely be "I don't really want to be tracked myself or have to track others" and all they think of is "We can dynamically allocate the resources based on where people are!".
My and many others first thought would likely be "I don't really want to be tracked myself or have to track others" and all they think of is "We can dynamically allocate the resources based on where people are!".
MIT Media Lab holds a discussion series around "Black Mirror" episodes. I think it's an interesting and engaging approach.
The MIT Media Lab is a community of creatives and technologists who may be active or future builders of the worlds that Black Mirror explores. This viewing and discussion series brings this community together to imagine and discuss technological futures and ethical implications, framed around the Black Mirror episodes we watch together. Each week we watch an episode and host a discussion facilitated by a researcher or practitioner whose work is pertinent to the episode.
https://www.media.mit.edu/posts/black-mirror/
The MIT Media Lab is a community of creatives and technologists who may be active or future builders of the worlds that Black Mirror explores. This viewing and discussion series brings this community together to imagine and discuss technological futures and ethical implications, framed around the Black Mirror episodes we watch together. Each week we watch an episode and host a discussion facilitated by a researcher or practitioner whose work is pertinent to the episode.
https://www.media.mit.edu/posts/black-mirror/
That is a neat idea! It illustrates the need for applied critical thinking, especially at the seams between concept / theory, and implementation / practice.
There is another aspect, though, which is the well-established and more formal instruction of experimental ethics. That is less of an analytic exercise like the MIT discussions, and more of a training for conducting responsible and ethical research. My opinion is the descriptive / discursive approach of the MIT discussions is insufficient to give that full understanding.
There is another aspect, though, which is the well-established and more formal instruction of experimental ethics. That is less of an analytic exercise like the MIT discussions, and more of a training for conducting responsible and ethical research. My opinion is the descriptive / discursive approach of the MIT discussions is insufficient to give that full understanding.
While I agree with you that ethics is important and that it's not taught sufficiently in STEM, it needs to be taught from a systemic point of view. It's a lie to teach we can be ethical in our current system where the profit motive is a principal factor in our decisions.
From my experience in STEM, the one ethics course I took was taught divorced from the political economy, which makes it sound like you can be ethical while trying to make profit. I think this is a fundamental inaccuracy.
From my experience in STEM, the one ethics course I took was taught divorced from the political economy, which makes it sound like you can be ethical while trying to make profit. I think this is a fundamental inaccuracy.
The idea that you can act ethically while still making a profit dates back to the Rennaisance - and I wouldn't say it's necessarily wrong. The idea that massive profits combined with accumulation of capital totally disproportionate to actual labour input when luck plays only a small role is a newer idea (and a newer phenomenon), though there are ethicists who don't see a problem with that either. I think some 19th C. German philosophers correctly decided not to pursue a moral route when they criticized capital accumulation.
I think the ethical problem, if there is one, can't lie with the simple fact of making profit (or the profit motive) - rather, it must lie either in how that profit came about, or how that profit is utilized. Most ethicists do not suppose an ethical problem with employment in itself, nor with reinvesting capital in itself.
I think the ethical problem, if there is one, can't lie with the simple fact of making profit (or the profit motive) - rather, it must lie either in how that profit came about, or how that profit is utilized. Most ethicists do not suppose an ethical problem with employment in itself, nor with reinvesting capital in itself.
I remember taking business ethics in college. The professor told us to keep a dirt file on our bosses in case we ever needed to retaliate. He got really upset when I challenged him
on it. I understood his point but I found it extremely ironic.
It is a useful tool especially if coupled with saved copies of weekly status reports of yours.
- Saved my butt a few times and often with no hard feelings when having had meekly pointed out the “dirt”.
- Saved my butt a few times and often with no hard feelings when having had meekly pointed out the “dirt”.
But if we take an ethics class, how can we justify spending 8-10 hours a day coming up with dark patterns to trick people to click on ads? /sarcasm
There's no evidence that the study of ethics increases ethical behavior.[1]
[1]https://qz.com/1582149/ethicists-are-no-more-ethical-than-th...
[1]https://qz.com/1582149/ethicists-are-no-more-ethical-than-th...
I would hardly characterize calling your mother or donating to charity as "ethical behavior".
Ethics don't need to be taught -- people willing to put ethics before other interests don't need to be taught how to think about things, and people who don't put ethics before their other interests aren't going to be persuaded by a 3 credit elective.
Ethics needs to be taught.
People have to encounter ethical challenges and consider how to respond to them before being put in the position of having to make a real ethical choice. Ethics isn't just about a gut check on "right vs wrong," and having a rational framework to apply to situations helps people reason about the choices they make.
And, let's make it mandatory, not elective.
People have to encounter ethical challenges and consider how to respond to them before being put in the position of having to make a real ethical choice. Ethics isn't just about a gut check on "right vs wrong," and having a rational framework to apply to situations helps people reason about the choices they make.
And, let's make it mandatory, not elective.
Why is this being downvoted? A younger colleague participated in such a course (because he was on an NIH-funded project), and it was case study after case study, together with "how would you have acted in such a situation, and why". He reported that he found it extremely useful.
People hear ethics and think moralizing; who wants to sit through that?
[deleted]
When I was in school ethics was taught. But, it was insufficient and it targeted not right or wrong but instead mostly legal liability.
This is where ethics becomes difficult because it depends on what you believe about the universe. For example, in the book Age of Surveillance Capitalism professors who look at people as herd animals is discussed. If people are just another herd animal and it's ok to herd them as we please (like how we do with cattle) than the ethical impact would be different from someone who believes there is a clear right or wrong way to treat people.
This impacts the ethics of using technology. Yet, it's often not discussed.
This is where ethics becomes difficult because it depends on what you believe about the universe. For example, in the book Age of Surveillance Capitalism professors who look at people as herd animals is discussed. If people are just another herd animal and it's ok to herd them as we please (like how we do with cattle) than the ethical impact would be different from someone who believes there is a clear right or wrong way to treat people.
This impacts the ethics of using technology. Yet, it's often not discussed.
What makes you think they don’t?
It’s required for accreditation.
Ethics classes will help enlighten you to what’s wrong and why. It won’t stop you from doing them.
It’s required for accreditation.
Ethics classes will help enlighten you to what’s wrong and why. It won’t stop you from doing them.
[deleted]
I think you may find this has a cultural connection.
cigaser(2)
I disagree. I went to school to learn how to solve problems, not somebody's subjective view of ethics. Let people who care about ethics take ethics courses if they really want to. Make corporations responsible for behaving ethically and then figure out how they want to handle it. As a private person, I live life by my own ethics and am not interested in spending my money to have somebody else lecture me on their ethics.
This misunderstanding of what is ethics is exactly why we need to be teaching it.
Ethics is not about the moral compass of an individual, but it is about the moral obligations of the professional community as a whole.
What is and isn't Ethical has largely already been discussed and decided. It is up to us developers to learn about it and apply it to our field.
Ethics is not about the moral compass of an individual, but it is about the moral obligations of the professional community as a whole.
What is and isn't Ethical has largely already been discussed and decided. It is up to us developers to learn about it and apply it to our field.
Indeed, these ethics courses are not supposed to turn students towards the narrow path of goodness, rather the purpose is to enable them to spot temptation when it comes, because it will.
>What is and isn't Ethical has largely already been discussed and decided.
I agree with your post, except on this point. Ethics is never 'decided' in the light of new axioms, new evidence, and debates on which principles apply to which cases. Morality is a matter of dialogue, and the idea that anybody (computer scientist or not) should accept without the possibility of debate any ethical point is tantamount to dogmatism. It is disastrous on two fronts:
* It does not respect the student as a moral reasoner in many cases; the student is not taught to reason beyond the level of application if they are only taught principles, and the student is not taught to reason at all if they are only taught scenarios and analogies
* It presents ethics as something "those other people" (i.e. academic philosophers) do, rather than something everyone does in dialogue with others. Such a presentation will not stand up if people (for non-moral, epistemic reasons) lose trust in that establishment.
When you say "largely" you seem to acknowledge this, but I'd go further and say that certain edge cases impact on the whole. To teach ethics largely as something 'decided' is nearly equivalent to teaching that the best language for task X is Y, and and that the best langugaes for most tasks have already been decided, and your only reasoning should be when to decide when to use language Y, never to interrogate the language, to interrogate the use cases, or to interrogate other possible principles.
Ethics isn't "anything goes", but it certainly isn't "listen to these principles. The only freedom you have is reasoning about when to apply them".
I agree with your post, except on this point. Ethics is never 'decided' in the light of new axioms, new evidence, and debates on which principles apply to which cases. Morality is a matter of dialogue, and the idea that anybody (computer scientist or not) should accept without the possibility of debate any ethical point is tantamount to dogmatism. It is disastrous on two fronts:
* It does not respect the student as a moral reasoner in many cases; the student is not taught to reason beyond the level of application if they are only taught principles, and the student is not taught to reason at all if they are only taught scenarios and analogies
* It presents ethics as something "those other people" (i.e. academic philosophers) do, rather than something everyone does in dialogue with others. Such a presentation will not stand up if people (for non-moral, epistemic reasons) lose trust in that establishment.
When you say "largely" you seem to acknowledge this, but I'd go further and say that certain edge cases impact on the whole. To teach ethics largely as something 'decided' is nearly equivalent to teaching that the best language for task X is Y, and and that the best langugaes for most tasks have already been decided, and your only reasoning should be when to decide when to use language Y, never to interrogate the language, to interrogate the use cases, or to interrogate other possible principles.
Ethics isn't "anything goes", but it certainly isn't "listen to these principles. The only freedom you have is reasoning about when to apply them".
> What is and isn't Ethical has largely already been discussed and decided
By who? You're appealing to a non-existent, homogeneous "community" when you say
> the moral obligations of the professional community as a whole.
To take specific examples, maybe one community considers it a moral obligation to advance technology and knowledge no matter what, even if it has uses some people don't like such as crypto or facial recognition. Apparently the big-tech and silicon valley communities think it's okay to police speech on their platforms, whereas professionals who value free speech would consider that unethical.
As an engineer I don't want my tuition going towards anything that isn't directly useful to me, or that aren't my free choice. Until there are ethics regulations or something that actually impact my professional life, I'm not interested in spending money on it.
By who? You're appealing to a non-existent, homogeneous "community" when you say
> the moral obligations of the professional community as a whole.
To take specific examples, maybe one community considers it a moral obligation to advance technology and knowledge no matter what, even if it has uses some people don't like such as crypto or facial recognition. Apparently the big-tech and silicon valley communities think it's okay to police speech on their platforms, whereas professionals who value free speech would consider that unethical.
As an engineer I don't want my tuition going towards anything that isn't directly useful to me, or that aren't my free choice. Until there are ethics regulations or something that actually impact my professional life, I'm not interested in spending money on it.
> I disagree. I went to school to learn how to solve problems, not somebody's subjective view of ethics
https://www.jstor.org/stable/378062?seq=1
https://www.jstor.org/stable/378062?seq=1
Yes! The idea that "ethics" should be taught in technical courses disrespects the autonomy of students, who paid good money to learn about a topic they are interested in, not to hear your propaganda. These students know where to find a course on ethics if they want one. Forcing your ethical views on a captive audience of students in a course on, say, computer architecture, is unethical.
For a horrible example of "ethics" instruction at Harvard, see the "ethics" module for CS 61: Systems Programming and Machine Organization, at
https://embeddedethics.seas.harvard.edu/classes/cs-61-2019-f...
An excerpt:
Imagine you are a student who is taking this course because you need the technical knowledge you think it covers, and then you are asked to discuss family emojis. I think you would be right to be disgusted with the instructor, disgusted with the CS department, and disgusted with Harvard University.
For a horrible example of "ethics" instruction at Harvard, see the "ethics" module for CS 61: Systems Programming and Machine Organization, at
https://embeddedethics.seas.harvard.edu/classes/cs-61-2019-f...
An excerpt:
Sample Class Activity: After being introduced to the concept of representational harm, students are presented with a slide containing the current set of ‘yellow’ emoji representing families of different kinds. In small groups, students discuss what kinds of families are left out from the current set and whether those omissions constitute representational harms.
Imagine you are a student who is taking this course because you need the technical knowledge you think it covers, and then you are asked to discuss family emojis. I think you would be right to be disgusted with the instructor, disgusted with the CS department, and disgusted with Harvard University.
Exactly what I mean. Unfortunately, I think a large number of people on the site lean towards something like idealism or imposing moral and ethical views on others, hence why comments like this get downvoted.
"Please provide to the public, in an expedited manner, all information necessary to identify all proposals of known-vulnerable code from any U of MN experiment. The information should include the name of each targeted software, the commit information, purported name of the proposer, email address, date/time, subject, and/or code, so that all software developers can quickly identify such proposals and potentially take remedial action for such experiments."
That sounds more than reasonable as a request to make amends.
The article said that finding all this code is a real problem. If UMN and the students involved are contrite that should be easy to fulfill.
That sounds more than reasonable as a request to make amends.
The article said that finding all this code is a real problem. If UMN and the students involved are contrite that should be easy to fulfill.
I actually disagree as it would expose which maintainers accepted these sorts of patches and that can have implications for their reputations and livelihood. Beyond that, this is just "Open Source software can be vulnerable to bad actors (News at 10)".
That's true to a point, but hiding that information is arguably (and in my estimation) worse.
"We better not look for other incidences of this nefarious behavior because it might create a small amount of collateral damage. Better to leave those patches unexamined."
"We better not look for other incidences of this nefarious behavior because it might create a small amount of collateral damage. Better to leave those patches unexamined."
UMN has already said they've opened an investigation?
There seems to be transparency value in having a different organization do the investigation than made the original judgment to approve the research.
Maybe there's an "internal affairs" equivalent that we'd trust, but this reads to me like "UMN made an error in approving this research but don't worry because UMN is now going to look into it."
Maybe there's an "internal affairs" equivalent that we'd trust, but this reads to me like "UMN made an error in approving this research but don't worry because UMN is now going to look into it."
Sure, that's entirely reasonable. But that's not what the Linux Foundation is demanding, apparently.
Is it your belief that maintainers should be viewed as automatons, impervious to mistakes, like how many view the service industry?
Maintainers are humans, flawed just like us all, good maintainers choose to accept and learn from their gaffes.
Maintainers are humans, flawed just like us all, good maintainers choose to accept and learn from their gaffes.
But let's also keep in mind these are humans, flawed and all, who were experimented on without permission. Ethical (and consensual) experiments would never reveal the identity of the participants and their exact reaction(s) under the experimental conditions. Just because the experiment started without any ethical considerations doesn't mean all ethical considerations should be ignored with respect to the "data" collected by the experiment.
No, my belief is the IRB would only waive consent if this sort of public disclosure is not allowed. Naming-and-shaming is not a valid research goal. Improper research results in censure and exclusion from the scientific corpus, not full public disclosure.
[deleted]
I don't believe the Linux kernel community would perceive the information in that way. Everyone is vulnerable to this kind of bad faith submission; gregkh and others seem to understand that.
Individual kernel maintainers should probably be able to contact the IRB to inquire whether they were personally enrolled in the study. The IRB may reply to that, or they might ask the study to notify all enrolled participants individually. Those individuals could then on their own publicly disclose the fact that they were enrolled. But I don't see why the general public has any need to know anything and it's unreasonable to make the demand.
I can't think of any way other than what the kernel maintainers are already doing. I doubt those involved actually tracked the information needed separately, and even if they did I wouldn't trust them to have tracked everything.
I love this tactic. To me, if this was an actually controlled experiment, the last step in the experiment would be to undo the bugs they intentionally created. If they did not, this would definitely be sabotage. Based on that, they should have no problem being able to identify what they did.
It's kind of a catch-22 for the UMN.
It's kind of a catch-22 for the UMN.
It's like running an experiment by adding a venom in water supply, with a possible later remedial action of offering an antidote.
They seemingly had some success with the first step, until this was duly noticed.
The experiment itself may be a bad idea, but it's a good, useful wake-up call.
They seemingly had some success with the first step, until this was duly noticed.
The experiment itself may be a bad idea, but it's a good, useful wake-up call.
Agreed. It's a cool idea. It's not really computer science as much as a penetration test involving humans and processes.
Of course without any semblance of prior consent it isn't quite sabotage but definitely outside the realm of ethical
Of course without any semblance of prior consent it isn't quite sabotage but definitely outside the realm of ethical
>without any semblance of prior consent it isn't quite sabotage
I don't follow. You're saying you're supposed to tell me you're messing with me in order for it to be considered sabotage? That doesn't make any sense, so you must mean something different than that. The entire point of sabotage is to do it under the noses of those you've infiltrated.
Edit: I think I re-read to get your meaning. You're saying without consent it's bad but not quite to the level of sabotage. Not sure if sabotage requires intent to cause harm, but they full well knew what they were doing was not good. While that might not have been enough to sink the ship, it sure wasn't trying to help it stay afloat.
I don't follow. You're saying you're supposed to tell me you're messing with me in order for it to be considered sabotage? That doesn't make any sense, so you must mean something different than that. The entire point of sabotage is to do it under the noses of those you've infiltrated.
Edit: I think I re-read to get your meaning. You're saying without consent it's bad but not quite to the level of sabotage. Not sure if sabotage requires intent to cause harm, but they full well knew what they were doing was not good. While that might not have been enough to sink the ship, it sure wasn't trying to help it stay afloat.
Indeed. If UMN can't tell how to undo the damage, then it is intentional sabotage, and should be prosecuted as such.
This post might give a hint on where to look to find who would probably benefit from sabotaging Linux security.
https://www.mail-archive.com/[email protected]/msg12...
https://www.mail-archive.com/[email protected]/msg12...
Fiction writers would have a hard time making something like this up. But at the same time, of course this is what happens. It's one of the oldest plays in the book to infiltrate the group you want to subvert, and then make changes from within so it is no longer a threat.
> The UMN has not said yet whether it will be withdrawn.
Regardless of UMN's decision, perhaps the IEEE should consider withdraw of this paper from their journal regardless, their acceptance criteria reads:
> Discuss steps taken to ensure that participants and others who might have been affected by an experiment were treated ethically and with respect.
> If a paper raises significant ethical and/or legal concerns, it might be rejected based on these concerns.
https://www.ieee-security.org/TC/SP2021/cfpapers.html#Ethica...
In their apology letter, the researchers seem to acknowledge they ran afoul of this requirement:
> As many observers have pointed out to us, we made a mistake by not finding a way to consult with the community and obtain permission before running this study; we did that because we knew we could not ask the maintainers of Linux for permission...
Regardless of UMN's decision, perhaps the IEEE should consider withdraw of this paper from their journal regardless, their acceptance criteria reads:
> Discuss steps taken to ensure that participants and others who might have been affected by an experiment were treated ethically and with respect.
> If a paper raises significant ethical and/or legal concerns, it might be rejected based on these concerns.
https://www.ieee-security.org/TC/SP2021/cfpapers.html#Ethica...
In their apology letter, the researchers seem to acknowledge they ran afoul of this requirement:
> As many observers have pointed out to us, we made a mistake by not finding a way to consult with the community and obtain permission before running this study; we did that because we knew we could not ask the maintainers of Linux for permission...
The paper was already withdrawn. https://www-users.cs.umn.edu/~kjlu/papers/withdrawal-letter....
[deleted]
While I applaud the initiative (not that they need my recognition),
I am somewhat afraid that such withdrawal would snuff out any further debate in the scientific community about proper ethical behavior and its delimitation ( which are admittedly fuzzy )
I am somewhat afraid that such withdrawal would snuff out any further debate in the scientific community about proper ethical behavior and its delimitation ( which are admittedly fuzzy )
> withdrawal would snuff out any further debate in the scientific community about proper ethical behavior
You have this exactly backward.
Far from "snuffing out" discussion of ethical behavior - the problem in the first place was that these researchers deliberately avoided the primary mechanism already in place to examine proper ethical behavior - their university's Institutional Review Board.
The withdrawal is not an attempt to silence the discussion regarding the ethics at stake. Quite to the contrary, the withdrawal itself is an explicit admission of ethical failings: "our study design was inappropriate: specifically, it involved conducting research on the Linux kernel community without obtaining appropriate consent and approval." They continue, further conceding the ethical debate "We are withdrawing the paper so that we do not benefit from an improperly conducted study ... [and] to prevent our misguided research method from being seen as a model for how to conduct studies in the future."
Nearly the entire issue at stake stems from the fact that these researchers did not properly use the Institutional Review Board already at their disposal. IRBs exist for the primary purpose of determining what proper ethical behavior IS, using existing widely-agreed-upon guidelines ("delimitation" in your terms) and judgment, and in ensuring such behaviors are followed. In particular, you normally must consult the IRB before any experiment is conducted on subjects. IRBs also serve as a primary party in the examination of any allegations of ethical misconduct.
The researchers in question only consulted their IRB after already conducting the research and publishing its abstract to twitter, for which they received "heated discussion and pushback," and were then forced to removed the abstract and apologize to their own IRB for causing "many confusions and misunderstandings" according to the linked article.
You have this exactly backward.
Far from "snuffing out" discussion of ethical behavior - the problem in the first place was that these researchers deliberately avoided the primary mechanism already in place to examine proper ethical behavior - their university's Institutional Review Board.
The withdrawal is not an attempt to silence the discussion regarding the ethics at stake. Quite to the contrary, the withdrawal itself is an explicit admission of ethical failings: "our study design was inappropriate: specifically, it involved conducting research on the Linux kernel community without obtaining appropriate consent and approval." They continue, further conceding the ethical debate "We are withdrawing the paper so that we do not benefit from an improperly conducted study ... [and] to prevent our misguided research method from being seen as a model for how to conduct studies in the future."
Nearly the entire issue at stake stems from the fact that these researchers did not properly use the Institutional Review Board already at their disposal. IRBs exist for the primary purpose of determining what proper ethical behavior IS, using existing widely-agreed-upon guidelines ("delimitation" in your terms) and judgment, and in ensuring such behaviors are followed. In particular, you normally must consult the IRB before any experiment is conducted on subjects. IRBs also serve as a primary party in the examination of any allegations of ethical misconduct.
The researchers in question only consulted their IRB after already conducting the research and publishing its abstract to twitter, for which they received "heated discussion and pushback," and were then forced to removed the abstract and apologize to their own IRB for causing "many confusions and misunderstandings" according to the linked article.
This is extremely relevant and neither mentioned nor linked to in the attached article.
You might want to post this as a top level comment instead of a reply so it gets the attention it deserves.
You might want to post this as a top level comment instead of a reply so it gets the attention it deserves.
The conference has promised a review of their decision making about this paper, I hope they will be transparent about such concerns if there were any voiced in the process.
"researchers appear to have been experimenting on people,"
Nice of the Kernel community to spell this out in no uncertain terms. The original apology did not acknowledge this fact.
Nice of the Kernel community to spell this out in no uncertain terms. The original apology did not acknowledge this fact.
[deleted]
"On the Feasibility of Stealthily Introducing Vulnerabilities in Open-Source Software via Hypocrite Commits": https://github.com/QiushiWu/QiushiWu.github.io/blob/main/pap...
Can someone please explain in layman terms exactly what happened and why it’s a bad thing? I’m having a hard time figuring out why a pound of flesh needs to be extracted from the University.
They were “researching” how feasible it is to submit insecure code. In other words, if can they slip some vulnerabilities past the maintainers. To me, it’s the same as sabotage. Not only is it unkind, but if it succeeded, many real people depending on that code could be negatively impacted.
The real problem is that it wasn't done with the informed consent of any kernel maintainers. The information gathered during the exercise is potentially quite valuable for informing policy and procedure around how code is reviewed and accepted, it just wasn't gathered in an ethical way.
There's a method for testing this sort of thing: pen-tests with consent from higher ups or "war games" with blue-team/red-team. It's not even a novel thing to do this properly.
Nope. Lets say IRB made the correct decision and demanded informed consent. Then informed consent must be requested from ALL participants, maintainers, and reviewers of the study. Higher up maintainers don't make that decision for others in this scenario, sorry. Novel or not, lets be completely honest with ourselves, it's not an easy experiment to frame properly.
Researchers associated with UMN intentionally submitted patches to the Linux kernel with vulnerabilities in them as an experiment. OSS maintainers shouldn't have to be the unwitting test subjects of experiments and have their time wasted (or worse, have vulnerabilities make it into their source code) so that researchers can write a paper about it.
Another key detail is that the experiment was carried out without having been submitted to or reviewed by the IRB.
I feel it should almost be mentioned: It seems to me that the students did this on the existing reputation of the University. The University had been contributing and were generally trusted. So maintainers were reviewing the code based on that reputation.
Until we have detailed methodology of the experience it does not seems to have been the case (it looked like they use randomized email address).
The only traceable bad patch that can be traced to the university was for one student that tried to coerce its way to patch acceptance, by invoking slander and accusing of other kind of despicable behavior from the reviewer. Which ignited the whole drama (although it started way before that, said student didn't help with the already delicate situation and gave public awareness to the drama).
The only traceable bad patch that can be traced to the university was for one student that tried to coerce its way to patch acceptance, by invoking slander and accusing of other kind of despicable behavior from the reviewer. Which ignited the whole drama (although it started way before that, said student didn't help with the already delicate situation and gave public awareness to the drama).
> However, we believe experiments on people without their consent is unethical, and likely involves many legal issues. People are an integral part of the software review and development process. The Linux kernel developers are not test subjects, and must not be treated as such.
> As it is, the Linux developers and committers are now burning time reviewing several hundred UMN Linux kernel patches.
> As it is, the Linux developers and committers are now burning time reviewing several hundred UMN Linux kernel patches.
Other comments have explained what happened, so I'll explain that the "pound of flesh" demanded appears to be quite reasonable - assistance with finding all of the known bad bits of code that were submitted as part of this project.
They gave an irb waiver they should not have. The research was clearly unethical and they , the irb, gave it a rubber stamp. It’s likely there is a systemic problem and that this was not an isolated failure.
[deleted]
Thank you to everyone who responded, I now understand the situation much better.
Is there a link to the actual letter of demand?
So kinda a quick summary given the available information that I've seen.
Back in August 2020 some research was performed looking at introducing vulnerabilities into the Linux Kernel.[0] The paper indicates that three patches were submitted via anonymous gmail accounts to the mailing list and were never committed. The reviewers were provided a proper patch upon accepting the vulnerable one and received explicit confirmation that the maintainers would not move forward with the vulnerable patch.
I'm not sure when exactly questions started being raised about that research. Though I first became aware of it in December. The discussion was mostly around the human involvement and led to the prepublication of the paper being removed and clarifications being issued.[1]
Fast-forward to April 2021. The patch seems to have kicked things off[2]. This was called out as being an impossible situation, and as being a "known-invalid patch" by Greg KH [3].
It appears that at least three patches by this same author introduced vulnerabilities[4] according to Leon Romanovsky. Though I don't have links to the specific patches.
Leading to U.Mn's ban from contributing to the kernel by Greg KH[5]
---------
What is in my opinion unclear at least to me is whether these more recent patches are actually in bad faith or just simply bad. The prevailing theory is that they are part of more research into introducing vulnerabilities. As already stated though, that research and its paper were done in August of 2020. The more recent commits, the official story from Kangjie Lu, Qiushi Wu, and Aditya Pakki[6] are that they are part of "a new project that aims to automatically identify bugs introduced by other patches". This does somewhat align with statements[7] made indicating that the commits were from a static analysis tool being researched which was made prior to this blowing up. Though I will note that the author of that patch was _not_ one of the authors of the apology letter, so may genuinely be unrelated.
This tool story was not believed by Greg KH[8].
And his take is the one that has gained a lot of adoption. That these patches were intentionally made in bad faith for another paper.
I will state that the newer patches that caused problems did _not_ follow the methodology that the original research followed to try to prevent vulnerabilities from actually being introduced into the repo. The original paper, while certainly had issues with methodology and experimenting on people inappropriately, did take steps to prevent any actual vulnerabilities from being committed, whereas the ones in question did not, and even made it to stable branches.
If the official story from U.Mn is true the commits should also have been noted as having been found by a tool, and followed the proper procedure for that, which they did not do. Though it does appear that the vast majority of the patches that were reverted were legitimate patches.[9] Atleast spot checking replies on that mailing list.
I mean on a whole the original research was questionable, but I kind of want to be more charitable in my interpretation of the more recent events but honestly that original patch that kicked things off is pretty bad.
[0] https://github.com/QiushiWu/QiushiWu.github.io/blob/main/pap...
[1] https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
[2] https://lore.kernel.org/linux-nfs/20210407001658.2208535-1-p...
[3] https://lore.kernel.org/linux-nfs/YH5%[email protected]...
[4] https://lore.kernel.org/linux-nfs/YH+zwQgBBGUJdiVK@unreal/
[5] https://lore.kernel.org/linux-nfs/[email protected]...
[6] https://lore.kernel.org/lkml/CAK8KejpUVLxmqp026JY7x5GzHU2YJL...
[7] https://lore.kernel.org/lkml/CAAa=b7dnrz5Pz5hMUc29VHJb9ucFkW...
[8] https://lore.kernel.org/lkml/YH%2FfM%[email protected]/
[9] https://lore.kernel.org/lkml/202104221451.292A6ED4@keescook/
Back in August 2020 some research was performed looking at introducing vulnerabilities into the Linux Kernel.[0] The paper indicates that three patches were submitted via anonymous gmail accounts to the mailing list and were never committed. The reviewers were provided a proper patch upon accepting the vulnerable one and received explicit confirmation that the maintainers would not move forward with the vulnerable patch.
I'm not sure when exactly questions started being raised about that research. Though I first became aware of it in December. The discussion was mostly around the human involvement and led to the prepublication of the paper being removed and clarifications being issued.[1]
Fast-forward to April 2021. The patch seems to have kicked things off[2]. This was called out as being an impossible situation, and as being a "known-invalid patch" by Greg KH [3].
It appears that at least three patches by this same author introduced vulnerabilities[4] according to Leon Romanovsky. Though I don't have links to the specific patches.
Leading to U.Mn's ban from contributing to the kernel by Greg KH[5]
---------
What is in my opinion unclear at least to me is whether these more recent patches are actually in bad faith or just simply bad. The prevailing theory is that they are part of more research into introducing vulnerabilities. As already stated though, that research and its paper were done in August of 2020. The more recent commits, the official story from Kangjie Lu, Qiushi Wu, and Aditya Pakki[6] are that they are part of "a new project that aims to automatically identify bugs introduced by other patches". This does somewhat align with statements[7] made indicating that the commits were from a static analysis tool being researched which was made prior to this blowing up. Though I will note that the author of that patch was _not_ one of the authors of the apology letter, so may genuinely be unrelated.
This tool story was not believed by Greg KH[8].
And his take is the one that has gained a lot of adoption. That these patches were intentionally made in bad faith for another paper.
I will state that the newer patches that caused problems did _not_ follow the methodology that the original research followed to try to prevent vulnerabilities from actually being introduced into the repo. The original paper, while certainly had issues with methodology and experimenting on people inappropriately, did take steps to prevent any actual vulnerabilities from being committed, whereas the ones in question did not, and even made it to stable branches.
If the official story from U.Mn is true the commits should also have been noted as having been found by a tool, and followed the proper procedure for that, which they did not do. Though it does appear that the vast majority of the patches that were reverted were legitimate patches.[9] Atleast spot checking replies on that mailing list.
I mean on a whole the original research was questionable, but I kind of want to be more charitable in my interpretation of the more recent events but honestly that original patch that kicked things off is pretty bad.
[0] https://github.com/QiushiWu/QiushiWu.github.io/blob/main/pap...
[1] https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
[2] https://lore.kernel.org/linux-nfs/20210407001658.2208535-1-p...
[3] https://lore.kernel.org/linux-nfs/YH5%[email protected]...
[4] https://lore.kernel.org/linux-nfs/YH+zwQgBBGUJdiVK@unreal/
[5] https://lore.kernel.org/linux-nfs/[email protected]...
[6] https://lore.kernel.org/lkml/CAK8KejpUVLxmqp026JY7x5GzHU2YJL...
[7] https://lore.kernel.org/lkml/CAAa=b7dnrz5Pz5hMUc29VHJb9ucFkW...
[8] https://lore.kernel.org/lkml/YH%2FfM%[email protected]/
[9] https://lore.kernel.org/lkml/202104221451.292A6ED4@keescook/
I have followed the situation somewhat closely as well and there seem to be a lot of misinformation being thrown around. As far as I can tell (correct me if I'm wrong), no actual malicious commit has been found to be deliberately introduced and merged into the tree, and most of the recent commits that triggered this whole mess were from a claimed custom static analyzers that resulted in wrong/useless patches but not necessarily malicious.
I honestly think the situation is somewhat overblown, and some maintainers think so as well. To quote Jason Gunthorpe:
> So, this revert is based on not trusting the authors to carry out their work in the manner they explained? From what I've reviewed, and general sentiment of other people's reviews I've read, I am concerned this giant revert will degrade kernel quality more than the experimenters did - especially if they followed their stated methodology.
and Doug Ledford:
> I have to agree with Jason. This seems like trying to push a thumbtack into a bulletin board using a pyle driver. Unless the researchers are lying (which I've not seen a clear indication of), the 190 patches you have selected here are nothing more than collateral damage while you are completely missing the supposed patch submission addresses from which the malicious patches were sent!
https://lore.kernel.org/lkml/20210421180155.GA2287172@nvidia...
https://lore.kernel.org/lkml/18edc472a95f1d4efe3ef40cc9b8d26...
I honestly think the situation is somewhat overblown, and some maintainers think so as well. To quote Jason Gunthorpe:
> So, this revert is based on not trusting the authors to carry out their work in the manner they explained? From what I've reviewed, and general sentiment of other people's reviews I've read, I am concerned this giant revert will degrade kernel quality more than the experimenters did - especially if they followed their stated methodology.
and Doug Ledford:
> I have to agree with Jason. This seems like trying to push a thumbtack into a bulletin board using a pyle driver. Unless the researchers are lying (which I've not seen a clear indication of), the 190 patches you have selected here are nothing more than collateral damage while you are completely missing the supposed patch submission addresses from which the malicious patches were sent!
https://lore.kernel.org/lkml/20210421180155.GA2287172@nvidia...
https://lore.kernel.org/lkml/18edc472a95f1d4efe3ef40cc9b8d26...
> As far as I can tell (correct me if I'm wrong), no actual malicious commit has been found to be deliberately introduced and merged into the tree, and most of the recent commits that triggered this whole mess were from a claimed custom static analyzers that resulted in wrong/useless patches but not necessarily malicious.
My understanding is that the malicious commits described in their paper were submitted under alias email addresses, and the authors have not identified those addresses or the commits. So at this point there is no way to confirm that these malicious commits were properly reverted besides taking the authors word for it. To quote Mike Dolan's letter: "While the U of MN researchers claimed to take steps to prevent inclusion of vulnerabilities in the final software, their failure to gain consent suggests a lack of care. There are also amplified consequences because Linux kernel changes are picked up by many other downstream projects that build off of the kernel codebase." And I think it's fair for maintainers to question the competence and want to be able to verify everything, especially considering the consequences if the authors made a mistake.
But yes, assuming they used fake accounts, none of those 190 or so patches selected are the malicious ones from the paper. None of them appear to introduce any vulnerabilities, and the same for the weird commits from Aditya.
My understanding is that the malicious commits described in their paper were submitted under alias email addresses, and the authors have not identified those addresses or the commits. So at this point there is no way to confirm that these malicious commits were properly reverted besides taking the authors word for it. To quote Mike Dolan's letter: "While the U of MN researchers claimed to take steps to prevent inclusion of vulnerabilities in the final software, their failure to gain consent suggests a lack of care. There are also amplified consequences because Linux kernel changes are picked up by many other downstream projects that build off of the kernel codebase." And I think it's fair for maintainers to question the competence and want to be able to verify everything, especially considering the consequences if the authors made a mistake.
But yes, assuming they used fake accounts, none of those 190 or so patches selected are the malicious ones from the paper. None of them appear to introduce any vulnerabilities, and the same for the weird commits from Aditya.
"But when you're charged with maintaining the most important operating system on the planet"
Correction, two planets.
Correction, two planets.
Can they prosecute UMN? A lot of people and businesses could have suffered extensive damages from their changes, and it seems like some kind of charge should hold up in court.
Should we all contact UMN to complain?
Edit: please explain why I’m being downvoted for asking sincere questions.
Should we all contact UMN to complain?
Edit: please explain why I’m being downvoted for asking sincere questions.
>Should we all contact UMN to complain?
For what purpose other than to harass UMN staff? It's obvious they're well aware of the issue and that the community isn't happy with their actions. They've got staff and students that read HN and twitter.
Calling them at this point doesn't accomplish much other than being a DDoS attack and shooting the messenger (there's no way you're going to get in contact with the people who actually conducted the study).
For what purpose other than to harass UMN staff? It's obvious they're well aware of the issue and that the community isn't happy with their actions. They've got staff and students that read HN and twitter.
Calling them at this point doesn't accomplish much other than being a DDoS attack and shooting the messenger (there's no way you're going to get in contact with the people who actually conducted the study).
You're probably being downvoted because "Should we all contact UMN to complain?" sounds like what twitter outrage mobs do.
Fair enough, but I’m not advocating a DDOS attack or anything like that. Just a phone call or email similar to how we might contact our representative.
This would effectively be a DDOS attack, whether or not you intend it as one.
If that’s the case, then although you can wish that all reps have huge staff to collect communications from their constituents, it’s not true, and telling people to call their reps would be even more DDOS because at least universities can act as a firewall. Your local reps in comparison will Never be as well resourced.
The purpose of calling reps is to air grievances, everyone involved understands this, and they almost certainly have other phone lines for more critical functions. In contrast, a university is contacted for all sorts of reasons related to research/study, and complaining en mass would be highly disruptive to this.
So the argument you lead with is about extra phone lines? Then one can expect that professors have a university office line, and universities have Way More competency and discretionary budget in hiring PR firms and in-house staff to deal with dynamic and acute events.
In terms of sheer muscle power, most public officials can't compare to the behemoth that is a university. They are even outclassed on agility.
It may be your local official's job to do something, but the reality of budget anemia is stiffer than their obligations to you.
In terms of sheer muscle power, most public officials can't compare to the behemoth that is a university. They are even outclassed on agility.
It may be your local official's job to do something, but the reality of budget anemia is stiffer than their obligations to you.
That's the problem with your suggestion. The authors of the papers did not mean to advocate to make human experiments either, yet they did. You suggestion would result in a DDoS attack anyway..
I'm giving you an upvote. Look around your house and count the number of linux kernels running. My count is 6 that I know about. I haven't seen the actual vulnerable code submitted to know how critical the vulnerabilities are but I believe these grad students are liable both civilly and criminally. Not advocating for mob justice but there needs to be more than a slap on the wrist. For those of us who live and breath software security everyday this is kind of a big deal.
I don't agree this is the way to fix the bigger problem, which is the acceptance that every commit to the kernel is done in good faith. As mentioned in another comment, I believe having these grad students to take a look at the ethical impacts of their research is a way forward. Another would be to somehow cast some blame into their supervisor, which should know more.
I understand that what they did was, and is bad and shouldn't be done. However how many other people do not also purposely submit buggy patches? In the end of the day, this happening just show vulnerabilities of the merging system itself.
I understand that what they did was, and is bad and shouldn't be done. However how many other people do not also purposely submit buggy patches? In the end of the day, this happening just show vulnerabilities of the merging system itself.
I think we disagree on the bigger problem. In my view the bigger problem is the erosion of trust for open source. Over the last 10-ish years there has been a flood of research and marketing around open source software security. I wrote a white paper about it back in 2011. We know there are risks in open source and we know vulnerabilities are created both intentionally and accidentally. We also know open source maintainers are overworked and human. There will be mistakes and we must prepare for them. This is the reason why the fine folks at Sonatype, Snyk, and WhiteSource have jobs.
These grad students wanted to make a splash and went after one of the most important code bases on the planet. It stopped being an ethical problem when the kernel maintainers had to manually search for vulnerabilities. They are using hours that could be used elsewhere. The Linux Foundation is paying Greg Kroah-Hartman to solve this problem, so they have a financial loss due to the actions of these grad students. There's your civil liability. They "knowingly cause(d) the transmission of a program, information, code, or command, and as a result of such conduct, intentionally causes damage without authorization, to a protected computer" so there's your criminal liability from the Computer Fraud and Abuse Act. There's probably criminal liability in the state where they live as well.
These grad students wanted to make a splash and went after one of the most important code bases on the planet. It stopped being an ethical problem when the kernel maintainers had to manually search for vulnerabilities. They are using hours that could be used elsewhere. The Linux Foundation is paying Greg Kroah-Hartman to solve this problem, so they have a financial loss due to the actions of these grad students. There's your civil liability. They "knowingly cause(d) the transmission of a program, information, code, or command, and as a result of such conduct, intentionally causes damage without authorization, to a protected computer" so there's your criminal liability from the Computer Fraud and Abuse Act. There's probably criminal liability in the state where they live as well.
The issue isn’t that every commit and committer is treated in good faith. They aren’t.
The issue is that U of Minnesota, and universities in general, had a good standing reputation with the Linux kernel group. Students at the University can still submit patches, but not currently with the strength of institutional credibility standing behind them.
Knowing who to trust and updating your trust when you’re wrong is part of healthy security.
The issue is that U of Minnesota, and universities in general, had a good standing reputation with the Linux kernel group. Students at the University can still submit patches, but not currently with the strength of institutional credibility standing behind them.
Knowing who to trust and updating your trust when you’re wrong is part of healthy security.
I'd argue yes. Linux is a dependency in a non-trivial subset of global infrastructure. If you tainted _without permission_ (implying no guidelines, no test scenario, no restrictions etc) your local water supply to see if you could and went with "It was a test! Here's some chlorine to fix it." you'd be in serious trouble. This study purposefully tainted the mainline kernel and in doing so absolutely everything that would be built on top of it. This was purposeful malicious action. Malicious actors should be punished.
Prosecute, no.
However, if this were in a more traditional scientific field, this sort of error would be treated as a serious lapse of experimental oversight protocols and the school or the participants would be sanctioned according to their professional discipline (in this case, that would probably be the IEEE).
Unethical experiments are a huge deal. If this had been an experiment on people face to face in e.g. a sociology context, then these students would likely have been expelled (and probably deported) and would likely be leaving the field.
Since it was in an IT area, the U's oversight rules probably weren't even applied.
However, if this were in a more traditional scientific field, this sort of error would be treated as a serious lapse of experimental oversight protocols and the school or the participants would be sanctioned according to their professional discipline (in this case, that would probably be the IEEE).
Unethical experiments are a huge deal. If this had been an experiment on people face to face in e.g. a sociology context, then these students would likely have been expelled (and probably deported) and would likely be leaving the field.
Since it was in an IT area, the U's oversight rules probably weren't even applied.
What extensive damages could anyone have suffered from their actions? Anyone can knowingly or unknowingly submit "hypocrite" patches. Beyond that each of the patches fixed a different already-existing security flaw and the research design included removing the hypocrite feature with a followup patch and maintainer education if the first were accepted.
As a graduate of the UMN these actions have reduced the value of my degree. Nobody should trust me because I'm probably tainted with the lack of ethics, and thus I'm unhireable. Fortunately I have a job that I'm proven in, but if they hit a downturn I might be in trouble.
> As a graduate of the UMN these actions have reduced the value of my degree.
Isn't that like saying an MIT Media Lab degree has reduced in value due to their relationship with Jeffrey Epstein, Nicky Negroponte and other deplorables.
Isn't that like saying an MIT Media Lab degree has reduced in value due to their relationship with Jeffrey Epstein, Nicky Negroponte and other deplorables.
>Isn't that like saying an MIT Media Lab degree has reduced in value due to their relationship with Jeffrey Epstein, Nicky Negroponte and other deplorables.
Yes, and this is also true.
Yes, and this is also true.
Sortof, but I'll bet you heard of MIT in a good way before, while this is the only time you have heard of the UMN. Thus they are tainted different
At least these "deplorables" weren't acting on behalf or with monetary contributions from the MIT Media Lab
Very utilitarian of you. I guess after you get your degree, it is up to you to show your worthiness..
Most likely because of the "suffered extensive damages", which does not seem to be supported by facts.
This is more a pride / trust issue than it is actual damages
This is more a pride / trust issue than it is actual damages
Can you explain? If I sneak in an exploit to widely used code, couldn’t the damages eventually be large once I or someone else takes advantage of it?
That is hypothetical, you are claiming companies have suffered damages, not that companies COULD suffer IF an exploit made it in.
Not trying to nit-pick, but your quote of what I said specifically left off my words "could have" before the word "suffered" so I do not think I claimed what you are claiming I claimed.
Isn't there such a thing as people being charged with conspiracy to commit XYZ?
Isn't there such a thing as people being charged with conspiracy to commit XYZ?
You misunderstand why syshum answered the way he did.
GP used the phrase "actual damages" which is a precise term of art in law. Damages have to exist (actual damages already suffered, or statutory damages imposed on conduct which has already transpired) to form a basis for a lawsuit -- the law does not concern itself with hypotheticals.
Whether or not there could be 'large damage' in the future (your "once I or someone else takes advantage") is irrelevant, immaterial, and only barely actionable. You could seek an injunction to attempt to prevent further potentially-damaging conduct. But you would not be able to claim any actual damages and would generally not be entitled to any form of compensation, not even for the attorney's fees generated in seeking the injunction.
As a sibling points out, conspiracy is question of criminal law, not civil law. Furthermore, in almost all jurisdictions within the United States, conspiracy requires at least one of the members of the conspiracy to have actually committed some overt act in furtherance of the crime. It should be impossible to find these researchers guilty of a conspiracy -- even if you claim that introducing hypocrite commits was the overt act, it is already clear that their intention is to academically investigate (and improve, if you're feeling charitable) the state of open-source security. Their actions (introducing hypocrite commits) are not in and of themselves violations of criminal law, so you'd still have to prove that they actually conspired to do something actually criminally illegal as well, e.g. intent to actually damage in some specific way, facilitated by these commits, some specific entity Foo which uses the linux kernel. It's perfectly clear that no such intent existed.
Whether or not there could be 'large damage' in the future (your "once I or someone else takes advantage") is irrelevant, immaterial, and only barely actionable. You could seek an injunction to attempt to prevent further potentially-damaging conduct. But you would not be able to claim any actual damages and would generally not be entitled to any form of compensation, not even for the attorney's fees generated in seeking the injunction.
As a sibling points out, conspiracy is question of criminal law, not civil law. Furthermore, in almost all jurisdictions within the United States, conspiracy requires at least one of the members of the conspiracy to have actually committed some overt act in furtherance of the crime. It should be impossible to find these researchers guilty of a conspiracy -- even if you claim that introducing hypocrite commits was the overt act, it is already clear that their intention is to academically investigate (and improve, if you're feeling charitable) the state of open-source security. Their actions (introducing hypocrite commits) are not in and of themselves violations of criminal law, so you'd still have to prove that they actually conspired to do something actually criminally illegal as well, e.g. intent to actually damage in some specific way, facilitated by these commits, some specific entity Foo which uses the linux kernel. It's perfectly clear that no such intent existed.
Conspiracy to X is a criminal charge, not the basis for a lawsuit. You can only sue someone over actual damages (or statutory damages), not hypothetical damages. If some DA wants to pursue criminal charges against UMN, that's separate from anyone asking to be made whole for the consequences of UMN's actions.
There's also no general "Conspiracy" modifier to crimes. Rather, "Conspiracy to X" is a separate law for only a handful of X.
There's also no general "Conspiracy" modifier to crimes. Rather, "Conspiracy to X" is a separate law for only a handful of X.
How is creating trust issues on a kernel not an actual damage?
Actual Damages is generally a legal term in which someone has to point to an actual dollar value of the damages they suffered. i.e John ran into my car, here is an estimate for $1,000 to repair the damages.
Trust is very hard to make an actual damages claim for, it can be done but outside of the Linux Foundation I am not sure what companies would have an actual damages claim
Trust is very hard to make an actual damages claim for, it can be done but outside of the Linux Foundation I am not sure what companies would have an actual damages claim
Actual Damages would be the time & effort to review previous patches for vulnerabilities, which is also the motivation for their demands from UMN. That's complicated by the fact that it's unpaid volunteer work, but a bill for the man-hours to evaluate the extent of the mess and clean it up seems like a clear-cut basis for establishing damages.
I have spent time doing contract development, and also made volunteer contributions to open source projects (not the Linux kernal, though). Certainly, if I were pulled into reviewing UMN kernel patches, I would be tracking my hours at my standard hourly rate, so that I would have the documentation for a claim for actual damages. Cleaning up other people's bad-faith crap is not what open source developers volunteer for. UMN caused real work for real people for bad-faith reasons, and should pay up. The work they caused is deflecting volunteers from otherwise productive activities.
I suspect many volunteer kernel developers do contract development to pay the bills -- it should not be hard for them to just create a billing code for UMN clean-up. The Linux foundation can aggregate them all and dump that on UMN.
(Also, I have a MSEE from UMN, and I can tell you that the next attempt at fund raising from me is not going to go well for them.)
I suspect many volunteer kernel developers do contract development to pay the bills -- it should not be hard for them to just create a billing code for UMN clean-up. The Linux foundation can aggregate them all and dump that on UMN.
(Also, I have a MSEE from UMN, and I can tell you that the next attempt at fund raising from me is not going to go well for them.)
Hopefully, the Linux Foundation or someone with standing sends the university a bill - if they don't have intrinsic motivation to behave ethically, then the force of the market is a powerful motivation.
Hypothetically let's say that is damage that they can be held liable for.
How far does it extend? The patches would have been reviewed and approved by someone at the linux foundation. Are they complicit and liable? Same goes for the person that merged the code. I don't think that's a door I want to open.
How far does it extend? The patches would have been reviewed and approved by someone at the linux foundation. Are they complicit and liable? Same goes for the person that merged the code. I don't think that's a door I want to open.
Maybe we’ll see a UMN clause added to popular open source licenses to combat this sort of behavior. Maybe it’ll be “don’t do this” or it’ll go as far as “don’t do this or your right to legally use this software is terminated immediately.”
[deleted]
A little remimder for all the people advocating for teaching "Ethics" to students/researchers: Linux Foundation is sponsored by Microsoft, Google, Intel etc.
Need a mirror to look at, anybody?
Need a mirror to look at, anybody?
[deleted]
IRB - Institutional Review Board
see https://slatestarcodex.com/2017/08/29/my-irb-nightmare/
If I were an organization with intent on profiting from a space where a competitor offered a free equivalent or better product, that had a religious like following that was rapidly gaining support, what could I do?
First, I could undermine the de facto leaders of those spaces with topical period appropriate accusations which, at the very least, limits their social power and thus their power to defend themselves and their groups.
Second, I can undermine the public's trust in what they offer.
Sure, call me paranoid. But this wouldn't be the first time this has happened.
First, I could undermine the de facto leaders of those spaces with topical period appropriate accusations which, at the very least, limits their social power and thus their power to defend themselves and their groups.
Second, I can undermine the public's trust in what they offer.
Sure, call me paranoid. But this wouldn't be the first time this has happened.
I understand your point as a thought experiment rather then a conspiracy theory in earnest so I don't think your down votes are justified.
However, I disagree with your second point. The way this is being handled has only increased my trust in the Linux Foundation.
However, I disagree with your second point. The way this is being handled has only increased my trust in the Linux Foundation.
It is nice, isn't it. I find their work inspirational.
People who don't understand IRBs are vastly overstating the IRB issue. Yes, they should have sought it in advance, but on the other hand almost nobody in computer science gets human subjects research training. IRBs issue waivers of consent fairly often, particularly if there's extremely low risk to participants and disclosing the research will significantly affect the results (both of which are true here). This particular study would very likely have been granted an apriori waiver of consent without much hassle at all.
Frankly, the study seems to have been fairly responsibly designed. They fixed actual kernel bugs, but included some additional more subtle latent bugs, if the subtle bugs were not detected, they corrected and removed the latent bugs and educated maintainers who accepted them.
Frankly, the study seems to have been fairly responsibly designed. They fixed actual kernel bugs, but included some additional more subtle latent bugs, if the subtle bugs were not detected, they corrected and removed the latent bugs and educated maintainers who accepted them.
> This particular study would very likely have been granted an apriori waiver of consent without much hassle at all.
We had significant IRB work to do for a grant that involved usability testing of an application, to determine if it was more effective than existing tools. The IRB committee wanted to know what kinds of observations about software usability would be recorded and what kind of analysis we would be doing. They wanted assurance testers would be anonymous. They wanted assurance all testers would know about the evaluation, and how the analysis would be used. They wanted assurance that evaluation of the software's usability would not morph into an evaluation of the testers themselves.
We had significant IRB work to do for a grant that involved usability testing of an application, to determine if it was more effective than existing tools. The IRB committee wanted to know what kinds of observations about software usability would be recorded and what kind of analysis we would be doing. They wanted assurance testers would be anonymous. They wanted assurance all testers would know about the evaluation, and how the analysis would be used. They wanted assurance that evaluation of the software's usability would not morph into an evaluation of the testers themselves.
Ok? Why would that qualify for a waiver? Disclosing the research to participants poses no risk to the study's integrity.
> but on the other hand almost nobody in computer science gets human subjects research training.
I took an undergrad CS class that specifically covered IRB processes and how to research human subjects.
If this isn't covered at that institution, when students/researchers are performing studies involving online communities, then that is a major oversight of UMN.
I took an undergrad CS class that specifically covered IRB processes and how to research human subjects.
If this isn't covered at that institution, when students/researchers are performing studies involving online communities, then that is a major oversight of UMN.
I agree it's an oversight, but I'm not sure it's unexpected. Some fields just don't typically do human subjects research so IRBs don't come up. I can't see why cryptographers would be wasting their time learning about how to prevent repeats of Tuskegee for example. So it probably depends on how these people got into the security research field.
I do think the IRB got this one wrong, but I am not surprised that they got it wrong.
From my experience with an IRB, they were most worried about researchers doing another Tuskegee experiment or a Stanford prison experiment. The question they were interested in was something like “are you going to torture/abuse/harm people directly?”
The people on the board have likely never thought about the issue of introducing a (possibly important) vulnerability into software which is widely used. That also has the potential for harm, but doesn’t look obvious to the IRB in the way that deliberately infecting someone with syphilis would.
Getting an exempt determination for this research would likely have been extremely easy at any IRB at any university in the country. Applications from CS departments are probably less than 1% of what they do (that figure is a guess so if anyone knows better please correct it!).
From my experience with an IRB, they were most worried about researchers doing another Tuskegee experiment or a Stanford prison experiment. The question they were interested in was something like “are you going to torture/abuse/harm people directly?”
The people on the board have likely never thought about the issue of introducing a (possibly important) vulnerability into software which is widely used. That also has the potential for harm, but doesn’t look obvious to the IRB in the way that deliberately infecting someone with syphilis would.
Getting an exempt determination for this research would likely have been extremely easy at any IRB at any university in the country. Applications from CS departments are probably less than 1% of what they do (that figure is a guess so if anyone knows better please correct it!).
In my experience, what an IRB is worried about varies wildly by IRB. My graduate university was very careful, to the point of frustration (I had a four email back-and-forth with them about a project with no human subjects). The institution I did a postdoc with on the other hand was fairly lax and hands off.
Current institution is somewhere in the middle.
Current institution is somewhere in the middle.
True in my experience. A four email back and forth about a project w/ no human subjects is a pain. I had several months of back and forth about simply using data on events which had happened 15 years earlier. (I made (what felt like to me like) appeals to metaphysics to argue that it was very unlikely my using this data could retroactively harm people in the study.)
I would be surprised if there are IRBs which get a lot of action from CS departments, and more surprised if they have members who can make a qualified evaluation of the costs and benefits of this research.
I'm also not convinced that it was a bad idea. This was extremely poorly executed, but illustrates an important (though maybe obvious) point: this method for introducing vulnerabilities would work.
A better way to handle it, IMO, would have been to involve someone at the very top in the Linux organization, to say "hey we are going to do this to see if it works - are you ok with that?" and then as soon as it works once, or on some defined, very small number of times, stop immediately and tell everyone what you did.
I would be surprised if there are IRBs which get a lot of action from CS departments, and more surprised if they have members who can make a qualified evaluation of the costs and benefits of this research.
I'm also not convinced that it was a bad idea. This was extremely poorly executed, but illustrates an important (though maybe obvious) point: this method for introducing vulnerabilities would work.
A better way to handle it, IMO, would have been to involve someone at the very top in the Linux organization, to say "hey we are going to do this to see if it works - are you ok with that?" and then as soon as it works once, or on some defined, very small number of times, stop immediately and tell everyone what you did.
The research plan is said to include reversing the hypocrite patch. So basically you're replacing one existing bug (if the researchers could find it, anyone else could) with an obscure bug (that at least initially only the researchers are aware of) and then only if the obscure bug slips through the maintainers, then later the obscure bug is also purged. So even if the IRB understands what's going on, I think there's a reasonable beneficence case.
> there's extremely low risk to participants
How do you plan to determine the potential risk for all kernel users in case the patch finds its way into a release? No IRB who understands this would ever approve of such "research".
> disclosing the research will significantly affect the results
Nothing a few 100% watertight patches as distraction and maybe cooperation from some maintainers couldn't fix to a reasonable degree. Either way that doesn't justify the experiment.
How do you plan to determine the potential risk for all kernel users in case the patch finds its way into a release? No IRB who understands this would ever approve of such "research".
> disclosing the research will significantly affect the results
Nothing a few 100% watertight patches as distraction and maybe cooperation from some maintainers couldn't fix to a reasonable degree. Either way that doesn't justify the experiment.
An IRB did in fact approve the research as part of the peer review publication process.
> Nothing a few 100% watertight patches as distraction and maybe cooperation from some maintainers couldn't fix to a reasonable degree. Either way that doesn't justify the experiment.
The extent to which a bad actor can use patching of bugs to introduce subtle vulnerabilities is actually valuable knowledge. We only know this even happened because it was disclosed by the researchers. Make no mistake any person or organization who wants to do this will have zero qualms about doing it.
> Nothing a few 100% watertight patches as distraction and maybe cooperation from some maintainers couldn't fix to a reasonable degree. Either way that doesn't justify the experiment.
The extent to which a bad actor can use patching of bugs to introduce subtle vulnerabilities is actually valuable knowledge. We only know this even happened because it was disclosed by the researchers. Make no mistake any person or organization who wants to do this will have zero qualms about doing it.
> An IRB did in fact approve the research as part of the peer review publication process.
The IRB did not approve the research, it declared that it was "exempt". That's essentially a determination that the research poses absolutely no risk of harm to anyone and does not require a detailed review, which seems like a very questionable decision to me. One of the things that's worth investigating is whether the researchers were honest in describing their experiment to the IRB.
Also, it's been alleged that the IRB exemption was only sought after the research was conducted and the paper was submitted for publication. If true, that seems to imply that they lied on their NSF grant application, because the principal investigator would have had to certify that either the research did not involve human subjects or that they had already applied for IRB approval/exemption.
The IRB did not approve the research, it declared that it was "exempt". That's essentially a determination that the research poses absolutely no risk of harm to anyone and does not require a detailed review, which seems like a very questionable decision to me. One of the things that's worth investigating is whether the researchers were honest in describing their experiment to the IRB.
Also, it's been alleged that the IRB exemption was only sought after the research was conducted and the paper was submitted for publication. If true, that seems to imply that they lied on their NSF grant application, because the principal investigator would have had to certify that either the research did not involve human subjects or that they had already applied for IRB approval/exemption.
Why is it a questionable decision? I do not see the risk it poses. Literally anyone including organizations far more well funded and vested than UMN are likely submitting hypocrite patches. At its worst, this research is merely substituting one existing bug with a new bug.
And yeah, approval was sought after the research was conducted. Which itself will result in corrective action (very likely: training in human subjects research).
And yeah, approval was sought after the research was conducted. Which itself will result in corrective action (very likely: training in human subjects research).
This was already a known risk. The gains from quantifying how bad the risk are is far outweighed by the fact that they introduced a new risk.
[deleted]
> IRBs issue waivers of consent fairly often, particularly if there's extremely low risk to participants and disclosing the research will significantly affect the results (both of which are true here).
Apparently, the unwilling test subjects seems to strongly disagree with your conclusion in particular (I would guess) when it comes to the first point.
Apparently, the unwilling test subjects seems to strongly disagree with your conclusion in particular (I would guess) when it comes to the first point.
Well I'm sure this will become an IRB case-study, but I really don't think anyone is going to care.
> if the subtle bugs were not detected, they corrected and removed the latent bugs and educated maintainers who accepted them.
Last I looked, this was claimed but unverified since they have yet to give any way to identify the patches in question.
Last I looked, this was claimed but unverified since they have yet to give any way to identify the patches in question.
Publicly disclosing the patches would actually harm the reputation of any maintainer who accepted them and expose them to harassment and potential employment issues, so I strongly suspect an IRB would not allow it. It's entirely possible to fix bugs and educate without disclosing that the bugs were deliberate or part of the research.
You're skipping the part where the IRB would have to approve injecting vulnerabilities into a public software project, a process which involves obfuscating and lying to humans.
Right, but according to what's said about the study design, they (1) fixed an existing bug
(2) introduced a new bug, and
if maintainers didn't notice the new bug, (3) followed up with a patch removing the new bug
Overall there's benefit in that an existing bug is removed and no introduced bugs persist. I don't see why IRB would be likely to object to this.
Overall there's benefit in that an existing bug is removed and no introduced bugs persist. I don't see why IRB would be likely to object to this.
"Yes, they should have sought it in advance, but on the other hand almost nobody in computer science gets human subjects research training."
This is a major failing.
This is a major failing.
It is appalling to my partner, a phd in bio research, that even as undergrads we didn't have at least a couple of courses covering what ethics was, how it informs, and how it applies to everything we do.
Worst of all, because of this lack of education, many big tech firms have done unethical experimentation on us unsuspecting users, simply because the developers never questioned if they should do it.