How I Fail – Ian Goodfellow(veronikach.com)
veronikach.com
How I Fail – Ian Goodfellow
http://www.veronikach.com/how-i-fail/how-i-fail-ian-goodfellow-phd14-computer-science/
41 comments
I think the most likely cause of his success is not related to his talent, but his focus. When I was choosing a college, I wasn’t thinking about advisors. It certainly never occurred to me to find the best guys in the field and try to get in under them. It just sounds too far fetched to be a real possibility. But it is a real possibility, of course, and the people that identify that and go for it have a remarkable outcome of accumulating advantage. It’s like compound interest. Go to the best school you can. Get under a top advisor in grad school. Publish in top journals. Work for major companies. At this point you’re surrounded by the best and have become the best. You have to know what you want early, and then optimize your ass off. I was late to the party, but the last couple years have been huge successes for me relative to previous parts of my life, and that started with saying to myself “start by applying to the very best and working down from there.”
Probably the failure I consider the biggest is that I spent most of my PhD trying to solve supervised learning for computer vision using unsupervised feature learning methods, and was caught totally off guard when Alex, Ilya, and Geoff won the ImageNet contest with purely supervised methods. I think that in general wasting time writing papers that turn out to be dead ends is the main way that I fail in my own eyes.
Wow, what an amazing story.
And worth pointing out that Goodfellows work on unsupervised feature learning is clearly seen in his invention of GANs.
Edit, and:
I think it’s hard to extract value from negative results in machine learning because it can be so hard to tell what caused the negative result. A negative result might point to something very fundamental wrong with an idea, but it might also just be the result of a very small software bug, the wrong idea of the hyperparameter values to try out, too small of a model, etc.
So, so true. I read comments on HN on how important negative results are in ML and I look at all the things I’ve tried and someone else made work, and all I see is reporting negative results just discourages ppl from trying things.
This is a great interview.
Wow, what an amazing story.
And worth pointing out that Goodfellows work on unsupervised feature learning is clearly seen in his invention of GANs.
Edit, and:
I think it’s hard to extract value from negative results in machine learning because it can be so hard to tell what caused the negative result. A negative result might point to something very fundamental wrong with an idea, but it might also just be the result of a very small software bug, the wrong idea of the hyperparameter values to try out, too small of a model, etc.
So, so true. I read comments on HN on how important negative results are in ML and I look at all the things I’ve tried and someone else made work, and all I see is reporting negative results just discourages ppl from trying things.
This is a great interview.
Good interview. It really surprised me to see a Stanford undergrad fail at anything to be honest, much less admission to a similarly ranked school for a specific PhD concentration they are known for.
I guess in this circumstance it actually was their loss.
I guess in this circumstance it actually was their loss.
I did my undergrad at Cambridge University and I was surprised at how normal people were. Of course, there were a couple of utter geniuses, but they were the exception.
I think an undergrad friend of mine put it best: "Cambridge doesn't take the top 1%, it takes a certain type of person from the top 10%"
I think an undergrad friend of mine put it best: "Cambridge doesn't take the top 1%, it takes a certain type of person from the top 10%"
How many of your friends had anything less than 3 A's (A* since 2010) at A-level?
There's 424,000 students got into university [0] and Cambridge enrols 3,480 out of 17,000 applicants [1]. So they don't take even 1% and only the top 4% of students even bother applying.
Edit: I suspect that 3,480 includes overseas students which makes the the percentage of UK enrolments is even less.
Further [1]:
> In 2006, 5,228 students who were rejected went on to get 3 A levels or more at grade A
Perhaps being surrounded by the top 1% makes you feel very normal/average, but there's certainly nothing normal about the Cambridge applicants and your figure of 10% definitely isn't correct.
I got accepted at Imperial with predictions of ABC (+ no extra curricular) to do Maths and Computer Science there's no way Oxford or Cambridge would have looked at me without 4 A's + a significant amount of extra curricular activities.
[0]: https://www.theguardian.com/education/live/2016/aug/18/a-lev...
[1]: https://en.wikipedia.org/wiki/University_of_Cambridge#Admiss...
There's 424,000 students got into university [0] and Cambridge enrols 3,480 out of 17,000 applicants [1]. So they don't take even 1% and only the top 4% of students even bother applying.
Edit: I suspect that 3,480 includes overseas students which makes the the percentage of UK enrolments is even less.
Further [1]:
> In 2006, 5,228 students who were rejected went on to get 3 A levels or more at grade A
Perhaps being surrounded by the top 1% makes you feel very normal/average, but there's certainly nothing normal about the Cambridge applicants and your figure of 10% definitely isn't correct.
I got accepted at Imperial with predictions of ABC (+ no extra curricular) to do Maths and Computer Science there's no way Oxford or Cambridge would have looked at me without 4 A's + a significant amount of extra curricular activities.
[0]: https://www.theguardian.com/education/live/2016/aug/18/a-lev...
[1]: https://en.wikipedia.org/wiki/University_of_Cambridge#Admiss...
> So they don't take even 1% and only the top 4% of students even bother applying.
I went to the college with the highest number of state school students. After participating in outreach events to get more people from a variety of backgrounds to apply to Cambridge I honestly think the biggest bottleneck is people from state schools not applying. They hear all the stories about strange interviews or being surrounding by posh people they have nothing in common with and just don't bother.
I was the only person in my (rather large) school year to get into Cambridge, and only a couple of people applied. I was told all sorts of strange things about the application process which turned out not to be true. I imagine it must be far easier to apply if you go to a school where almost everyone does and the staff are well versed in what to get ready.
I went to the college with the highest number of state school students. After participating in outreach events to get more people from a variety of backgrounds to apply to Cambridge I honestly think the biggest bottleneck is people from state schools not applying. They hear all the stories about strange interviews or being surrounding by posh people they have nothing in common with and just don't bother.
I was the only person in my (rather large) school year to get into Cambridge, and only a couple of people applied. I was told all sorts of strange things about the application process which turned out not to be true. I imagine it must be far easier to apply if you go to a school where almost everyone does and the staff are well versed in what to get ready.
When my son interviewed there one of the tutors told me he thought they should abandon the interviewing and select at random from a pool of the top 10%.
It really surprised me to see a Stanford undergrad fail at anything to be honest
Why? The impression I've gotten from a lot of time in industry is that the average Stanford student/graduate is, well, average.
Why? The impression I've gotten from a lot of time in industry is that the average Stanford student/graduate is, well, average.
If you are Indian you need a 1500+ + 4.0 + good extra curriculars and there is a very good chance you are not getting in. It might similar for other folks, but I don't know. So if the people getting into Stanford are average not sure who you are comparing them to.
You probably need better than a 1500 these days. Closer to 1600. For what it's worth, I got a 1500 out of 1600 with a 4.0 and went to my safety (My EC's were nonexistent and never applied to Stanford though..)
Because Stanford is incredibly selective, and is one of the 4 premier CS institutions in the world?
I dunno, as someone that went to a public school with a 50% accept rate it seems difficult to imagine “academic superhumans” struggle with anything, ever.
I dunno, as someone that went to a public school with a 50% accept rate it seems difficult to imagine “academic superhumans” struggle with anything, ever.
I'm sure you believe that there's something special about Stanford. Many people believe that.
What I'm saying is that, in my experience, the belief doesn't hold up. If it were possible to reliably identify "academic superhumans" and concentrate them in one place, we'd have absolutely indisputable evidence of it. The fact that we don't suggests that perhaps your beliefs about how Stanford's admission practices correlate with quality of graduates may be mistaken.
What I'm saying is that, in my experience, the belief doesn't hold up. If it were possible to reliably identify "academic superhumans" and concentrate them in one place, we'd have absolutely indisputable evidence of it. The fact that we don't suggests that perhaps your beliefs about how Stanford's admission practices correlate with quality of graduates may be mistaken.
[deleted]
The best performing students I've ever met weren't ridiculously smart. They were slightly above average, but they excelled at finding out exactly what the teacher was going to test and drilling that ruthlessly. They also worked harder than most of their peers.
They also know how to present their answers well, and I think teachers subconsciously give them more credit when they're not exactly right. As a funny anecdote, once I let a top copy my homework because he had just come home from a family emergency. He actually got a higher score on that homework than I did, despite the content of our answers being the same.
They also know how to present their answers well, and I think teachers subconsciously give them more credit when they're not exactly right. As a funny anecdote, once I let a top copy my homework because he had just come home from a family emergency. He actually got a higher score on that homework than I did, despite the content of our answers being the same.
> it seems difficult to imagine “academic superhumans” struggle with anything, ever.
I know a person who went to a comparable institution in my country, who was considered something of a prodigy by his peers (and me) both before being selected and after joining the institution, and who struggled a lot with his academic responsibilities. In fact, I know a few of them. One of them had to drop out of the institution after repeatedly trying to finish his academic programme, and after two years of extensions.
People struggle with things, in general. Everybody has their struggles, everybody has their demons. Even "superhumans". What we should be aiming for is not the lack of struggles — for that is merely the absence of effort — but perseverance in the face of it and growth from going through it.
I know a person who went to a comparable institution in my country, who was considered something of a prodigy by his peers (and me) both before being selected and after joining the institution, and who struggled a lot with his academic responsibilities. In fact, I know a few of them. One of them had to drop out of the institution after repeatedly trying to finish his academic programme, and after two years of extensions.
People struggle with things, in general. Everybody has their struggles, everybody has their demons. Even "superhumans". What we should be aiming for is not the lack of struggles — for that is merely the absence of effort — but perseverance in the face of it and growth from going through it.
I forget who, but in (99% sure) Werner Herzig's Lo and Behold there's a (Stanford?) professor who says when they first did some MOOC they had all the kids in the Stanford course taking it as well, and of the 50,000 or whatever number of people taking the course, the top Stanford student finished like #430 in the course.
[deleted]
That was a nice read, thanks!
In general, I don't know how I feel about these articles, they seem to be trending among my Twitter friends. I know that these interviews are meant as an encouragement. To show that even the great people of our field had to struggle, so we should not give up when we are still struggling.
However, these stories can also be understood negatively - if you didn't make it, you probably did not work or try hard enough. What makes this effect stronger is the selection bias - we do not see articles of the other 99% that work their tails of and do not become big in their fields. With a society that focuses on being the best, 'making it', etc. it leaves a lot of people feel inadequate, incompetent, or even depressed, even when they are big net contributors to society.
E.g. think of the 50-year old professor a moderately successful academic career. He/she is educating hundreds of students and having a big positive impact on their lives. But they are seen by others and themselves as failed academics who were not good enough to make it.
(I don't know what the solution is.)
In general, I don't know how I feel about these articles, they seem to be trending among my Twitter friends. I know that these interviews are meant as an encouragement. To show that even the great people of our field had to struggle, so we should not give up when we are still struggling.
However, these stories can also be understood negatively - if you didn't make it, you probably did not work or try hard enough. What makes this effect stronger is the selection bias - we do not see articles of the other 99% that work their tails of and do not become big in their fields. With a society that focuses on being the best, 'making it', etc. it leaves a lot of people feel inadequate, incompetent, or even depressed, even when they are big net contributors to society.
E.g. think of the 50-year old professor a moderately successful academic career. He/she is educating hundreds of students and having a big positive impact on their lives. But they are seen by others and themselves as failed academics who were not good enough to make it.
(I don't know what the solution is.)
Well there isn't a solution because the metrics for "failure" and "success" are fuzzy. In my mind, someone is a "success" if they accomplish the challenge of being the person they see themselves as/want to be. It's way harder than it sounds.
So while there are certainly some people who, as Pickard said "made no mistakes but still lost," there are others who didn't put in sufficient effort and failed as a result.
I'd imagine that there are more in the latter camp than in the former. In your example, the professor is only a failure if their goal was something other than moderately successful academic.
So while there are certainly some people who, as Pickard said "made no mistakes but still lost," there are others who didn't put in sufficient effort and failed as a result.
I'd imagine that there are more in the latter camp than in the former. In your example, the professor is only a failure if their goal was something other than moderately successful academic.
I suspect the whole goal thing is a problem, setting some specific outcome you expect to reach when you have no idea what obstacles you might face on the way is setting yourself up for failure.
Indeed! Goal setting is one of the most complex tasks we do.
Doing it successfully, defined as a measurable outcome, requires that you have a pretty accurate assessment of both the landscape, your abilities and the path to achieve the goal - with some factor considered for deviation from the charted path.
Depending on the scope of goal, the confidence interval on probability of success probably decreases exponentially with respect to time.
Doing it successfully, defined as a measurable outcome, requires that you have a pretty accurate assessment of both the landscape, your abilities and the path to achieve the goal - with some factor considered for deviation from the charted path.
Depending on the scope of goal, the confidence interval on probability of success probably decreases exponentially with respect to time.
Doing it successfully, defined as a measurable outcome, requires that you have a pretty accurate assessment of both the landscape, your abilities and the path to achieve the goal - with some factor considered for deviation from the charted path.
I agree, but for longer term goals, an extremely large amount of luck involved. Sometimes the right opportunities and persons just line up. I often realize that I'd be in a completely different (and probably worse) place if my 22 and 26-year old self didn't encounter the right persons and opportunities.
My current career path was largely unplanned, but is better than I hoped for or planned when I was in my early to mid-20ies.
I agree, but for longer term goals, an extremely large amount of luck involved. Sometimes the right opportunities and persons just line up. I often realize that I'd be in a completely different (and probably worse) place if my 22 and 26-year old self didn't encounter the right persons and opportunities.
My current career path was largely unplanned, but is better than I hoped for or planned when I was in my early to mid-20ies.
>(I don't know what the solution is.)
Is it possible that stories like this are supposed to affect everyone differently?
-#1 Take it as encouragement => succeed, and be "happier", and share the success.
-#2 Take it as encouragement => Fail (and possibly move on to something more appropriate for you, and you are all the wiser for it)
-#3 Take it negatively => Quit at something and move on to something else. (see #2)
-#4 Take it negatively => Succeed anyways despite the negativity (see #1)
Maybe I missed an idea here, but how could it hurt anyone to share this information?
-#1 Take it as encouragement => But fail excessively the rest of your life?
-#2 Take it negatively => Give up and never try again?
It seems the positive outcomes outweigh the negative. And the negative outcomes don't seem reasonable to blame on an article.
Is it possible that stories like this are supposed to affect everyone differently?
-#1 Take it as encouragement => succeed, and be "happier", and share the success.
-#2 Take it as encouragement => Fail (and possibly move on to something more appropriate for you, and you are all the wiser for it)
-#3 Take it negatively => Quit at something and move on to something else. (see #2)
-#4 Take it negatively => Succeed anyways despite the negativity (see #1)
Maybe I missed an idea here, but how could it hurt anyone to share this information?
-#1 Take it as encouragement => But fail excessively the rest of your life?
-#2 Take it negatively => Give up and never try again?
It seems the positive outcomes outweigh the negative. And the negative outcomes don't seem reasonable to blame on an article.
[deleted]
[deleted]
I feel the story of his professor who responded “why do you have an ‘a’ in my class?” Is so incredibly relatable. It seems that sometimes; it’s far to easy to do in some “luck”.
I'm getting a 404 Not Found. Anyone else?
... That's how he fails.
I'm the owner of the website (just made an account but @consp can confirm). Basically my webhosting provider didn't like all the traffic despite me having bought extra bandwidth. I'm hoping to get it fixed tomorrow
In the meanwhile here is the post: (link: https://web.archive.org/web/20180505190115/http://www.veroni...) web.archive.org/web/2018050519…
In the meanwhile here is the post: (link: https://web.archive.org/web/20180505190115/http://www.veroni...) web.archive.org/web/2018050519…
It has been down for a couple of hours.
I mentioned that in a comment here and, for reasons I don't understand, was immediately downvoted. I deleted the comment in case there is some sort of HN 404 taboo I'm not aware of.
I mentioned that in a comment here and, for reasons I don't understand, was immediately downvoted. I deleted the comment in case there is some sort of HN 404 taboo I'm not aware of.
> I mentioned that in a comment here and, for reasons I don't understand, was immediately downvoted. I deleted the comment in case there is some sort of HN 404 taboo I'm not aware of.
It's good to reflect on whether a comment that you posted that is downvoted really is problematic in some way, and if you see a real problem with that reflection to delete or edit it, but you shouldn't delete a comment when you don't understand a downvote because, frankly, lots of good posts get stray early downvotes, and that usually gets corrected over time.
It's good to reflect on whether a comment that you posted that is downvoted really is problematic in some way, and if you see a real problem with that reflection to delete or edit it, but you shouldn't delete a comment when you don't understand a downvote because, frankly, lots of good posts get stray early downvotes, and that usually gets corrected over time.
> I deleted the comment in case there is some sort of HN 404 taboo I'm not aware of.
Never do this. If someone doesn't like your comment, then you are just letting them win by removing it. That's exactly what they want: for the comment not to exist. Don't give them the satisfaction; people who engage in censorship practices don't deserve it.
Never do this. If someone doesn't like your comment, then you are just letting them win by removing it. That's exactly what they want: for the comment not to exist. Don't give them the satisfaction; people who engage in censorship practices don't deserve it.
From the top level site:
> This account has been suspended. > Either the domain has been overused, or the reseller ran out of resources.
> This account has been suspended. > Either the domain has been overused, or the reseller ran out of resources.
Getting a 404, the top level site has the error
"This account has been suspended. Either the domain has been overused, or the reseller ran out of resources."
Here's an archive link
https://web.archive.org/web/20180505190115/http://www.veroni...
"This account has been suspended. Either the domain has been overused, or the reseller ran out of resources."
Here's an archive link
https://web.archive.org/web/20180505190115/http://www.veroni...
The 404 is caused by the provider expecting less get requests than GB's bought. The author (sitting next to me) actually bought more traffic but the company decided to just stop after a number of GET requests.
Oh I thought the 404 was the joke.
The author clearly needs a new provider.
[deleted]
The Deep Learning Book by Goodfellow et al. is amazing. I found the writing to be very down-to-earth and it explains concepts without too much jargon. http://www.deeplearningbook.org/
I can only assume his answer to question 11 is in response to the very public discourse between him and Jürgen Schmidhuber.