The coder's worst nightmare (2016)(quora.com)
quora.com
The coder's worst nightmare (2016)
https://www.quora.com/What-is-a-coders-worst-nightmare/answer/Mick-Stute?share=1
50 comments
That's what I was thinking while reading this article, +1!
In short, the premise of the talk is: if you add a self replicating change to any self-hosted compiler:
- You can 'teach' the compiler to do things that are never explicitly stated in their source code, like, in this article, making the compiler modify programs that meet specific predicates.
- You can not trust that your compiler hasn't been modified without bootstrapping it yourself. You can't trust a program without trusting its compiler; and since compilers are programs, well it's turtles all the way down.
To build a trusted compiler, you have to bootstrap one yourself from some trusted manner, usually from hand-written assembly. In C, this would be akin to writing a small C compiler in assembly, using that to compile tcc, using that to compile a slightly larger C compiler, and that to compile clang or gcc.
But in other more complex languages, such as Rust, bootstrapping is not as clear cut. There's the Reproducible Builds project, but afaict there exists no minimal small Rust compiler you can bootstrap the latest one from: You have to go back the the original OCaml one and compile up from there, or just trust that the publicly available build is secure (which it most likely is).
I guess the lesson is that it's possible to inject hidden self-replicating behavior into any significantly complex system.
In short, the premise of the talk is: if you add a self replicating change to any self-hosted compiler:
- You can 'teach' the compiler to do things that are never explicitly stated in their source code, like, in this article, making the compiler modify programs that meet specific predicates.
- You can not trust that your compiler hasn't been modified without bootstrapping it yourself. You can't trust a program without trusting its compiler; and since compilers are programs, well it's turtles all the way down.
To build a trusted compiler, you have to bootstrap one yourself from some trusted manner, usually from hand-written assembly. In C, this would be akin to writing a small C compiler in assembly, using that to compile tcc, using that to compile a slightly larger C compiler, and that to compile clang or gcc.
But in other more complex languages, such as Rust, bootstrapping is not as clear cut. There's the Reproducible Builds project, but afaict there exists no minimal small Rust compiler you can bootstrap the latest one from: You have to go back the the original OCaml one and compile up from there, or just trust that the publicly available build is secure (which it most likely is).
I guess the lesson is that it's possible to inject hidden self-replicating behavior into any significantly complex system.
You can use mrustc to not have to go back to the original OCaml.
Hmm, that's an interesting project! You'd still have to bootstrap a cpp compiler first, though
Sure, you always have to bootstrap something.
An assembler can also do this same attack. You have to hand-assemble your basic assembler to machine code. In theory a sophisticated-enough hex editor could also be subverted, silently adding in code when it's used to edit executable-looking files. The system dynamic linker can be compromised to subvert any executable. Etc.
If a machine is compromised, it can't be recovered without access to a known-good machine.
If a machine is compromised, it can't be recovered without access to a known-good machine.
https://github.com/certik/bcompiler/ is an effort to do this very bootstrapping.
I get the theory here, but never having written a compiler myself, I’m curious how easy/difficult it would be to come up with those predicates in practice?
See 'Reflections on Rusting Trust' by Manish Goregaokar (In Pursuit of Laziness).
https://manishearth.github.io/blog/2016/12/02/reflections-on...
He modifies rustc with a predicate that replaces the string 'hello world' with something else. The method used to replicate the change is basically a self-modifying quine.
https://manishearth.github.io/blog/2016/12/02/reflections-on...
He modifies rustc with a predicate that replaces the string 'hello world' with something else. The method used to replicate the change is basically a self-modifying quine.
Coincidentally, Computerphile just posted this today:
https://www.youtube.com/watch?v=SJ7lOus1FzQ
https://www.youtube.com/watch?v=SJ7lOus1FzQ
Every time I read these stories I wonder if they arent, just that, stories, then I realize it would be kinda fun to create one of these lowlevel compiler infecting things, and realize nah there are atleast a few out there written for shits and giggles. If nothing else, just because of the stories.
The best way to do a modern rendition would be to couple it with some sort of infrastructure vulnerability like a package-squatting attack or similar, so it has a chance to infiltrate for at least a little while before getting killed.
The point would be to have it simply persist - and ideally for a little while too - but do nothing useful other than for example allocate an additional string on a local stack somewhere or something.
Absolute best case scenario would be to announce its existence one year after successful infiltration. :D
The point would be to have it simply persist - and ideally for a little while too - but do nothing useful other than for example allocate an additional string on a local stack somewhere or something.
Absolute best case scenario would be to announce its existence one year after successful infiltration. :D
> Several days later, an AT&T tech shows up with a disk and loads the proper compile and linker source and we recompile the compiler from the source.
Because he would bill for less time if he just showed up with the compiler binaries and copied them over the rogue ones?
"Let's take pristine source across the air-gap, and manipulate it on the infected computer where there is nonzero risk that the wrong compiler could somehow compile it."
What if the kernel, C library or utilities in /bin and /usr/bin had been involved too?
Because he would bill for less time if he just showed up with the compiler binaries and copied them over the rogue ones?
"Let's take pristine source across the air-gap, and manipulate it on the infected computer where there is nonzero risk that the wrong compiler could somehow compile it."
What if the kernel, C library or utilities in /bin and /usr/bin had been involved too?
So tl;dr
- the original creator created a self-replicating virus that would modify this one program, making it unmodifiable.
- actually it was in the compiler. woop. can't fix it without external fixes
- creates a backdoor into the system, could have fucked with the whole campus' data
- he would have gotten away with it too, if not for the fact that he made this supremacist text
So basically, dang.
- the original creator created a self-replicating virus that would modify this one program, making it unmodifiable.
- actually it was in the compiler. woop. can't fix it without external fixes
- creates a backdoor into the system, could have fucked with the whole campus' data
- he would have gotten away with it too, if not for the fact that he made this supremacist text
So basically, dang.
Holy crap. That went way deeper than I expected it to.
About halfway through, I was ready to copy the source to another computer and compile it there instead, and then suggest they nuke the original computer.
Turns out I was right. I just didn't know how right I was, which could be important for whatever they plan to do to the person who hacked that computer.
About halfway through, I was ready to copy the source to another computer and compile it there instead, and then suggest they nuke the original computer.
Turns out I was right. I just didn't know how right I was, which could be important for whatever they plan to do to the person who hacked that computer.
Given the computer model involved, this is probably in the late 80s. If you've read The Cuckoo's Egg, it suggests that getting law enforcement to take computer intrusion seriously was very hard back then.
I suppose the university could impose some kind of sanctions such as revoking the student's degree.
I suppose the university could impose some kind of sanctions such as revoking the student's degree.
>I was hired by a psychologist to fix a program that seemed to have "strange output" written by one of his ex-grad students.
>printed some white supremacy message
So I'm missing some details here maybe, but isn't that easier to ask the ex-student? He's probably responsible for all of this right? I don't see how anyone could get away with such thing.
>printed some white supremacy message
So I'm missing some details here maybe, but isn't that easier to ask the ex-student? He's probably responsible for all of this right? I don't see how anyone could get away with such thing.
Considering this was malicious sabotage, I doubt the ex-student was even in contact with his former advisor, and regardless a conversation probably wouldn't be very helpful.
That seems unlikely. From my reading there was a program (a simple research tool to ask people psychology questions), that was written by an "ex-Grad Student". Presumably it worked fine for some time, and then started exhibiting strange behaviour (after all if it was written to do this from the get go who would ever have used it?).
If we are going to do detective guess-work (#) get the list of students who took the test. This level of effort to poison a research lab suggests to me the questions raised some uncomfortable issues with one of the respondents, who happened to have compiler abilities and the ability to read the root password on the post-it note on professors desk.
(#) which frankly is how most detectives dive in - that one seems suspicious, look for evidence.
If we are going to do detective guess-work (#) get the list of students who took the test. This level of effort to poison a research lab suggests to me the questions raised some uncomfortable issues with one of the respondents, who happened to have compiler abilities and the ability to read the root password on the post-it note on professors desk.
(#) which frankly is how most detectives dive in - that one seems suspicious, look for evidence.
This level of dedication to white supremacy is unsurprising to me after spending years on the internet, yet there are so many people who refuse to believe it's a real problem.
It is easy to mistake 14-16 year olds with a lot of time on their hands for dedicated real supremacists. The best example was hordes of teenagers teaching a Microsoft chat bot to be a supremacist. Clearly they were having fun.
It's their way of rebelling against the establishment by trolling. Since a teenager can no longer troll by assuming a leftist position, they must do so by pretending to be far right.
It's their way of rebelling against the establishment by trolling. Since a teenager can no longer troll by assuming a leftist position, they must do so by pretending to be far right.
I don't think teenagers being racist is a normal activity that is whimsical fun. There are actual consequences to people. Do you think the black 12 to 16 year old with an interest in technology continues that interest when the chat bot starts yelling racial slurs at him and telling him his life his worthless. It's very easy for people who have never experienced racism to say he should just brush it off. "It was just a joke bro" is also one of the favorite tools of neo nazis and other groups to gauge interest from other people while being able to claim they aren't actually racist.
The story is also about a former grad student so he is well into adult hood but probably started off as one of those teenagers just "trolling".
The story is also about a former grad student so he is well into adult hood but probably started off as one of those teenagers just "trolling".
This specific story aside, the point is that for those in their teens, it's very possible that expressing those views is being done to "rebel" rather than as a true belief or as an opinion held with the full appreciation of its implications. i.e. the act of holding such a rejected view is the point.
Which isn't to say such views should be tolerated, merely that the approach taken for interacting with such individuals really should account for that.
Which isn't to say such views should be tolerated, merely that the approach taken for interacting with such individuals really should account for that.
>I don't think teenagers being racist is a normal activity that is whimsical fun.
Well you're wrong. Allow me to lay this out as thoroughly as I can: Teenagers love being edgy, i.e. they love outrage for the sake of outrage. Hitler is outrageous (and well-known), thus he features prominently in "rebellious" humor. There's nothing wrong with this, because the premise of the joke rests precisely ON the fact that associated atrocities are inappropriate, or perhaps the logic is absurd. The jokes only work because Hitler, or more generally racism, or whatever, is wrong, furthermore that the general consensus is indeed that is is wrong. In fact, the more you kvetch, the funnier it is (to a teenager).
Now, perhaps modern "jokes" have you confused into thinking that comedy is just a smug-and-response between some wank and the audience, but this is not so. Personally I want grit, I want ambiguity, I want the full range of human experience out of life. Humor is many things, but it is at least a way of getting a handle on the unsavory inevitabilities of life. If you want to succumb to a fear-based ideology and spend your life constructing an unlivable, bureaucratic tangle of anti-boogeyman legislation, go ahead, but do the humans a favor and leave us out of it.
>There are actual consequences to people.
No, there aren't. No child is being unknowingly radicalized by inappropriate jokes (except maybe as a response to the twits taking up arms against them).
Well you're wrong. Allow me to lay this out as thoroughly as I can: Teenagers love being edgy, i.e. they love outrage for the sake of outrage. Hitler is outrageous (and well-known), thus he features prominently in "rebellious" humor. There's nothing wrong with this, because the premise of the joke rests precisely ON the fact that associated atrocities are inappropriate, or perhaps the logic is absurd. The jokes only work because Hitler, or more generally racism, or whatever, is wrong, furthermore that the general consensus is indeed that is is wrong. In fact, the more you kvetch, the funnier it is (to a teenager).
Now, perhaps modern "jokes" have you confused into thinking that comedy is just a smug-and-response between some wank and the audience, but this is not so. Personally I want grit, I want ambiguity, I want the full range of human experience out of life. Humor is many things, but it is at least a way of getting a handle on the unsavory inevitabilities of life. If you want to succumb to a fear-based ideology and spend your life constructing an unlivable, bureaucratic tangle of anti-boogeyman legislation, go ahead, but do the humans a favor and leave us out of it.
>There are actual consequences to people.
No, there aren't. No child is being unknowingly radicalized by inappropriate jokes (except maybe as a response to the twits taking up arms against them).
Most teenagers are rebellious in that they don't want to listen to their parents, drink and try drugs, stay out past curfew
Most teenagers aren't advocating that Hitler had good ideas and that all black people should be lynched. Just because you have experience with these racists (hopefully you weren't one as a teenager) does not mean its ok to pretend that its fine to normalize this behavior
It used to be considered "normal" to bully and ruin the lives of gay people as a teenager too. Now you'll be hard pressed to find anybody under eighteen that would willingly do that to a classmate. Society has changed and shitty behavior like racism, antisemitism, and homophobia get called out as they should.
When they don't you end up with groups like the proud boys and closet white supremacist like the grad student in the article.
Most teenagers aren't advocating that Hitler had good ideas and that all black people should be lynched. Just because you have experience with these racists (hopefully you weren't one as a teenager) does not mean its ok to pretend that its fine to normalize this behavior
It used to be considered "normal" to bully and ruin the lives of gay people as a teenager too. Now you'll be hard pressed to find anybody under eighteen that would willingly do that to a classmate. Society has changed and shitty behavior like racism, antisemitism, and homophobia get called out as they should.
When they don't you end up with groups like the proud boys and closet white supremacist like the grad student in the article.
>Most teenagers aren't advocating that Hitler had good ideas and that all black people should be lynched.
You're right, they aren't.
>the proud boys
ah yes, the infamous multi-cultural white supremacist group who marry black women
You're right, they aren't.
>the proud boys
ah yes, the infamous multi-cultural white supremacist group who marry black women
[deleted]
>>There are actual consequences to people.
>No, there aren't. No child is being unknowingly radicalized by inappropriate jokes (except maybe as a response to the twits taking up arms against them).
OP already provided an example of consequences. Not radicalization of white kids but racial trauma to black kids.
>No, there aren't. No child is being unknowingly radicalized by inappropriate jokes (except maybe as a response to the twits taking up arms against them).
OP already provided an example of consequences. Not radicalization of white kids but racial trauma to black kids.
> I don't think teenagers being racist is a normal activity that is whimsical fun.
Bullies think bullying is fun. Spectators think the event is fun. The bullied one pays the price (don't ask me how I know).
Some bullies never grow. They think it's fun, it's a way of expressing themselves.
Yes, neither racism nor bullying is normal, but exhibitor of these behaviors normalize it.
Bullies think bullying is fun. Spectators think the event is fun. The bullied one pays the price (don't ask me how I know).
Some bullies never grow. They think it's fun, it's a way of expressing themselves.
Yes, neither racism nor bullying is normal, but exhibitor of these behaviors normalize it.
I think most black teenagers have a much thicker skin than you think. Which wouldn't surprise me because most of the outrage about racism seems to be coming from preemptive actions from progressive whites rather than the ones who are supposedly affected by it.
When I was a black teenager I got lucky in that I didn't encounter much racism until much later in my late teens when I was well on the way to wanting a job in tech. I know others who encountered it earlier and lost interest.
Interesting hypothesis, why not take the bus down to your local high school and test it out?
implying what, blacks are violent savages incapable of discussion? Get a life, you vile racist
Or possibly merely that high schoolers are incompletely socialised adolescents? Your own comments, e.g. "Teenagers love being edgy" show that you know this.
No need for the rebound accusation. If you're a teenager trying to be "edgy" it's not working.
Source: I went to a high school.
No need for the rebound accusation. If you're a teenager trying to be "edgy" it's not working.
Source: I went to a high school.
A teenager's temptation to make a thing (created by adults) do things the adult would not want ... is enormous. It is an inversion of power.
Some adults also feel powerless, and act like teenagers when given the opportunity.
If your focus is on the consistent operation of your systems (computer systems, social systems, etc) then you'll view their actions as attacks.
Sometimes there are real consequences, to be sure. Often the actions originate as a joke or a tweak of sorts, from people who have undeveloped concepts of empathy or responsiblity.
I don't think this is solvable. Moral absolutism is appealing in theory, but in practice, the effects are negative for all parties.
Of course there's a line which, when crossed, requires some kind of countervailing reaction. I don't know how to define that line, and I don't particularly trust others to do so either. Harm is real, but complicated.
A lot of damage can be done in the grey area between "offended" and "injured".
Some adults also feel powerless, and act like teenagers when given the opportunity.
If your focus is on the consistent operation of your systems (computer systems, social systems, etc) then you'll view their actions as attacks.
Sometimes there are real consequences, to be sure. Often the actions originate as a joke or a tweak of sorts, from people who have undeveloped concepts of empathy or responsiblity.
I don't think this is solvable. Moral absolutism is appealing in theory, but in practice, the effects are negative for all parties.
Of course there's a line which, when crossed, requires some kind of countervailing reaction. I don't know how to define that line, and I don't particularly trust others to do so either. Harm is real, but complicated.
A lot of damage can be done in the grey area between "offended" and "injured".
[deleted]
they exist offline too. it's a very disturbing trend.
14-16 year olds who are actively participating in white supremacist movements are white supremacists.
The person being targeted by their harassment campaign can't know and doesn't care that they're minors or they think they're kidding or whatever.
The harm is the same and the right-wing radicalization of young white men online is an extremely serious problem. People in this age range have killed people for this idealogy!
The person being targeted by their harassment campaign can't know and doesn't care that they're minors or they think they're kidding or whatever.
The harm is the same and the right-wing radicalization of young white men online is an extremely serious problem. People in this age range have killed people for this idealogy!
Give 'em 5-10 years, and it's a lot harder to tell them apart.
Reminds me of the "Robin Hood and Friar Tuck" hack.
Excerpts:
> They dug around in the operating-system listings and devised a thoroughly devilish set of patches. These patches were then incorporated into a pair of programs called 'Robin Hood' and 'Friar Tuck'. Robin Hood and Friar Tuck were designed to run as 'ghost jobs' (daemons, in Unix terminology); they would use the existing loophole to subvert system security, install the necessary patches, and then keep an eye on one another's statuses in order to keep the system operator (in effect, the superuser) from aborting them.
> They found the bandit ghost jobs running, and killed them... and were once again surprised. When Robin Hood was gunned, the following sequence of events took place:
> Finally, the system programmers did the latter - only to find that the bandits appeared once again when the system rebooted!
Quoted in the Jargon File: http://www.catb.org/jargon/html/meaning-of-hack.html
Original story: https://www.cs.utah.edu/~elb/folklore/xerox.txt
Excerpts:
> They dug around in the operating-system listings and devised a thoroughly devilish set of patches. These patches were then incorporated into a pair of programs called 'Robin Hood' and 'Friar Tuck'. Robin Hood and Friar Tuck were designed to run as 'ghost jobs' (daemons, in Unix terminology); they would use the existing loophole to subvert system security, install the necessary patches, and then keep an eye on one another's statuses in order to keep the system operator (in effect, the superuser) from aborting them.
> They found the bandit ghost jobs running, and killed them... and were once again surprised. When Robin Hood was gunned, the following sequence of events took place:
id1: Friar Tuck... I am under attack! Pray save me!
id1: Off (aborted)
id2: Fear not, friend Robin! I shall rout the Sheriff of Nottingham's men!
id1: Thank you, my good fellow!
> Finally, the system programmers did the latter - only to find that the bandits appeared once again when the system rebooted!
Quoted in the Jargon File: http://www.catb.org/jargon/html/meaning-of-hack.html
Original story: https://www.cs.utah.edu/~elb/folklore/xerox.txt
A couple things I wonder there:
How does Robin (id1) manage to print the "I am under attack" message? Shouldn't a kill be immediate?
Couldn't they just do something like sending a SIGSTOP to processes, or use killall in a tight loop? I get it's not Unix, but it seems reasonable similar concepts should exist.
Also, this is more in line with Ken Thompson's "Reflections on Trusting Trust"
How does Robin (id1) manage to print the "I am under attack" message? Shouldn't a kill be immediate?
Couldn't they just do something like sending a SIGSTOP to processes, or use killall in a tight loop? I get it's not Unix, but it seems reasonable similar concepts should exist.
Also, this is more in line with Ken Thompson's "Reflections on Trusting Trust"
You can put a signal handler in your program to catch the SIGSTOP and run some code before it gets killed.
You can't catch SIGSTOP or SIGKILL. If you're trying to eliminate a rogue process, you wouldn't as nice as to ask it to politely shut down. It could just refuse. So if you're killing it forcefully, it shouldn't get to print anything. If you're asking it politely, then it can just refuse to die and doesn't need a monitor process to restart it.
In this story this isn't Unix but I imagine the same should apply. If you're trying to terminate something with extreme prejudice, it shouldn't get any chance to print anything.
In this story this isn't Unix but I imagine the same should apply. If you're trying to terminate something with extreme prejudice, it shouldn't get any chance to print anything.
Can't catch those signals, but under a very specific sets of circumstances, they could be... not quite ignored, but basically, that is behavior.
Things which are pending completion of I/O won't die as soon as SIGTERM or SIGKILL are sent to them. If you have ever tried to kill a process which had open file handles on a stale NFS mount, you have experienced this behavior. Causing it to happen deliberately is a lot more challenging, as well as not having a process actually be BLOCKED due to I/O, but it was[1] doable.
[1] haven't tried to do this in about a decade, so won't speak about the currency of this approach, but back then, with some clever assembly and a kernel module, it was doable.
Things which are pending completion of I/O won't die as soon as SIGTERM or SIGKILL are sent to them. If you have ever tried to kill a process which had open file handles on a stale NFS mount, you have experienced this behavior. Causing it to happen deliberately is a lot more challenging, as well as not having a process actually be BLOCKED due to I/O, but it was[1] doable.
[1] haven't tried to do this in about a decade, so won't speak about the currency of this approach, but back then, with some clever assembly and a kernel module, it was doable.
Maybe they used a heartbeat?
Sure, many things could be done for two processes to monitor each other. The confusing bit is this happening at all:
id1: Friar Tuck... I am under attack! Pray save me!
[deleted]
I almost feel like that's too commonplace to be noteworthy anymore? There are lots of annoying programs I try to kill only for them to be restarted.
Did this sort of read like a writing-prompt? By day 4 he could have unraveled a larger ploy involving lizard people.
Putting the ‘Code’ into DaVinci Code.
Putting the ‘Code’ into DaVinci Code.
[deleted]
https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...