A Year at Google Brain (2020)(debugmind.com)
debugmind.com
A Year at Google Brain (2020)
https://www.debugmind.com/2020/01/04/paths-to-the-future-a-year-at-google-brain/
98 comments
It sounds like you've had a very different PhD experience than I did. Perhaps a difference in fields? My PhD in physics was probably the most humbling experience I've had in my professional life. I went from thinking I knew everything to learning that I knew nothing. To me that's one of the most important parts of a research education, to become aware of what you don't know and be provided tools to try figuring it out.
The most disconcerting part for me was coming back to society, realizing that no one actually knows what's going on most of the time, even when they think they do. I wish more people had the same humbling experience I did (not saying, by the way, that we need more PhD students, just the similar experience).
It's curious why you'd think that the only reason for doing a PhD is to contribute faster. Contribute faster to what? The author says he's having a lot of fun with his PhD because he gets to work on topics that are not immediately tied in with Googles business model.
This is exactly why we need PhD students doing fundamental research, because no one can see the immediate benefit. Because on the surface it has no value at all - so who else would look at it if not curious PhD students with little to no financial incentives.
Constraining possible research breakthroughs to what someone thinks is valuable a-priori, seems very naive and counter productive to me.
The most disconcerting part for me was coming back to society, realizing that no one actually knows what's going on most of the time, even when they think they do. I wish more people had the same humbling experience I did (not saying, by the way, that we need more PhD students, just the similar experience).
It's curious why you'd think that the only reason for doing a PhD is to contribute faster. Contribute faster to what? The author says he's having a lot of fun with his PhD because he gets to work on topics that are not immediately tied in with Googles business model.
This is exactly why we need PhD students doing fundamental research, because no one can see the immediate benefit. Because on the surface it has no value at all - so who else would look at it if not curious PhD students with little to no financial incentives.
Constraining possible research breakthroughs to what someone thinks is valuable a-priori, seems very naive and counter productive to me.
I had a ton of fun doing my PhD, but I’m pretty sure I’ll have had similar fun without it as well. The reason I suggested not doing a PhD is the forces at play in various angles (the whole pyramid scheme, academia politics, publish vs. perish, good luck going tenure track, insane hours, abusive practices, the whole luck based system of it all).
If you went through your entire PhD without seriously questioning your life choices at all then we definitely did a different PhD for sure! I and my colleagues constantly discussed the pros and cons and why the system is the way it is.
In general I can never present the problems as clearly as Hamming did so I’ll let him give my viewpoint since I agree with everything here - http://www.paulgraham.com/hamming.html
If you went through your entire PhD without seriously questioning your life choices at all then we definitely did a different PhD for sure! I and my colleagues constantly discussed the pros and cons and why the system is the way it is.
In general I can never present the problems as clearly as Hamming did so I’ll let him give my viewpoint since I agree with everything here - http://www.paulgraham.com/hamming.html
I'm now in my third year of my PhD and enjoying it more than ever, so there's that at least!
In all seriousness, I totally acknowledge that the variance in satisfaction with a PhD is very large. I'm lucky to have a great advisor and incredible labmates; my happiness with the program is largely due to them.
In all seriousness, I totally acknowledge that the variance in satisfaction with a PhD is very large. I'm lucky to have a great advisor and incredible labmates; my happiness with the program is largely due to them.
> I'm lucky to have a great advisor and incredible labmates; my happiness with the program is largely due to them.
This. So much this. As in just about every profession, people make or break places. Your advisor and lab mates are everything... choose wisely, and choose them vs a school for its name.
This. So much this. As in just about every profession, people make or break places. Your advisor and lab mates are everything... choose wisely, and choose them vs a school for its name.
As others have noted, it can be harder to move around academia than in industry. Your immediate community matters a lot more, whether you're a PhD student, a postdoc, or faculty. When things work it's great; when they don't, it's correspondingly worse.
> But ignoring that one persons warning about PhD still sounded like he had the typical arrogance contained by every person who ever did a PhD (including me). If someone says “don’t do it” and your reaction is to not even follow up with questions perhaps you should acknowledge you already know why they’re saying that!
One my favourite quotes
> “I never allow myself to have an opinion on anything that I don’t know the other side’s argument better than they do.” — Charlie Munger
One my favourite quotes
> “I never allow myself to have an opinion on anything that I don’t know the other side’s argument better than they do.” — Charlie Munger
> It’s still not clear to me a person of this cadre need a PhD in a technical field in this day and age; they might contribute just as much faster if they chose any other route.
They might be able to contribute technically but they might not be given the opportunity to do so if they don't have a PhD.
They might be able to contribute technically but they might not be given the opportunity to do so if they don't have a PhD.
In hard sciences that might be true but I doubt it’s necessary in tech - see George Hotz.
Hotz is kind of doing his own thing at comma.ai. If I founded my own company, I'm sure they would hire me for anything that I applied for.
> It’s still not clear to me a person of this cadre need a PhD in a technical field in this day and age
Hey there, thanks for sharing your experience.
Would you mind telling us which technical field? And which country?
I know several PhD students / recent grads that pursued this qualification primarily because of "academic inflation" - that in order to compete for top-paying jobs in the EU, the more qualified you are the better, and having "just a Masters degree" is no longer good enough.
I should add, these are also ex-pats, who left their developing country in pursuit of greener pastures. But it seems to be the case that getting a PhD - as terrible as the experience might be - can also help with career prospects, purely because one has a PhD (irrespective of the details of their thesis topic).
Whereas back home (in said developing country), it's the complete opposite. Qualifications hardly matter - or they certainly don't matter as much - only experience does. And given the thin supply of skilled labour, business owners have the mindset of "we'll take what we can get".
So I do think the choice to pursue a PhD can be an indirectly strategic move depending on your job prospects - and less about the actual topic you plan to research.
Just my 2 cents from a developing country that is experiencing excessive brain-drain! :)
Hey there, thanks for sharing your experience.
Would you mind telling us which technical field? And which country?
I know several PhD students / recent grads that pursued this qualification primarily because of "academic inflation" - that in order to compete for top-paying jobs in the EU, the more qualified you are the better, and having "just a Masters degree" is no longer good enough.
I should add, these are also ex-pats, who left their developing country in pursuit of greener pastures. But it seems to be the case that getting a PhD - as terrible as the experience might be - can also help with career prospects, purely because one has a PhD (irrespective of the details of their thesis topic).
Whereas back home (in said developing country), it's the complete opposite. Qualifications hardly matter - or they certainly don't matter as much - only experience does. And given the thin supply of skilled labour, business owners have the mindset of "we'll take what we can get".
So I do think the choice to pursue a PhD can be an indirectly strategic move depending on your job prospects - and less about the actual topic you plan to research.
Just my 2 cents from a developing country that is experiencing excessive brain-drain! :)
I also moved from India to the US to do a PhD in biomedical engineering! In the end I left the field and took up a tech job after my PhD because it became clear that this is the more sensible career choice. Given I put in 8 years of some of my most productive years into the PhD, I think I’d have been a much different place in tech if I had just made the jump earlier. The PhD helped but not nearly as much as the same number of years of actual experience in tech itself.
Of course if you can’t or won’t program and want to be in a hard science a PhD might be necessary but it’s become generally annoyingly competitive and fruitlessly bureaucratic to be in the field and depend on it for long term guaranteed career growth. When I said tech I meant software-related: as long as you are interested or willing to be there it still seems the best place to make money without too much sweat and not need a PhD to do it.
This also applies only if you’re good but not cream of the crop. If you’re doing a PhD under a superstar and have nature papers then you’re clearly set up well, but anything lower than that it’s not worth the competition and pain you need to put in to make a good career either in academia or industry.
Of course if you can’t or won’t program and want to be in a hard science a PhD might be necessary but it’s become generally annoyingly competitive and fruitlessly bureaucratic to be in the field and depend on it for long term guaranteed career growth. When I said tech I meant software-related: as long as you are interested or willing to be there it still seems the best place to make money without too much sweat and not need a PhD to do it.
This also applies only if you’re good but not cream of the crop. If you’re doing a PhD under a superstar and have nature papers then you’re clearly set up well, but anything lower than that it’s not worth the competition and pain you need to put in to make a good career either in academia or industry.
>aimed at commodity phd students in regular universities
Woof.
Woof.
It's an important reality to understand.
I did my PhD in an in-demand field at a top uni.
My partner did their PhD in a less in-demand field at a not so top uni.
The difference in our experiences -- and how those experiences were directly impacted by the financial motivations of the department/university -- was beyond staggering. My partner was cheap teaching labor. I was a source of fellowship+grant money and potentially valuable patents/prestige. I could leave and 10x my salary with a single email (ultimately did). My partner would've been lucky to teach high school. In literally every interaction with the university, you could tell.
Universities are just another org with budget and headcount and metrics to optimize. Academia is no longer sheltered from the shitty labor politics of finance capitalism. NEVER let anyone tell you otherwise.
I did my PhD in an in-demand field at a top uni.
My partner did their PhD in a less in-demand field at a not so top uni.
The difference in our experiences -- and how those experiences were directly impacted by the financial motivations of the department/university -- was beyond staggering. My partner was cheap teaching labor. I was a source of fellowship+grant money and potentially valuable patents/prestige. I could leave and 10x my salary with a single email (ultimately did). My partner would've been lucky to teach high school. In literally every interaction with the university, you could tell.
Universities are just another org with budget and headcount and metrics to optimize. Academia is no longer sheltered from the shitty labor politics of finance capitalism. NEVER let anyone tell you otherwise.
"I could leave and 10x my salary with a single email (ultimately did). My partner would've been lucky to teach high school."
That has more to do with the field than the school. A CS grad student at a third-tier school can leave and take a well-compensated corporate job whenever they want. A history grad student at a top-tier school will find that more difficult.
That has more to do with the field than the school. A CS grad student at a third-tier school can leave and take a well-compensated corporate job whenever they want. A history grad student at a top-tier school will find that more difficult.
> Academia is no longer sheltered from the shitty labor politics of finance capitalism.
It never was. And politics in general, not just labor.
It never was. And politics in general, not just labor.
I think prior to the 2000s academia was much less MBAified than it is now.
What does one do if we've been deemed not good enough for top schools or institutions?
Can you get a great advisor at a mid-tier place and a strong guarantee that you won't have to TA a ton? If not, opt out of academia.
I had a half dozen tenure-track offers and decided on industry. Honestly, academia is not what it used to be. Lots of professors come close to killing their under-paid grad students so that they can spend 20% of their time on blue sky research ideas. I get that time and then some without throwing poor kids under the bus. Without even asking, really. And make 3x what I would've made as a slave driver to boot.
Academia sucks, especially if you want to do good research. Seriously, outside of tippy-top elite institutions that can get the very best and brightest phd students, colleges and universities are some of the absolute worst places to do research these days.
And if you want to teach CS, don't bother with a PhD. Just ad junct a bit on the side of an industry gig, do not-insanely-bad, and you'll be full time within a year or two. If you want. Which you very likely wont, because teaching one class on the side while engineering is a lot more fun than teaching full time. Teaching is exhausting, even for people who love teaching.
I had a half dozen tenure-track offers and decided on industry. Honestly, academia is not what it used to be. Lots of professors come close to killing their under-paid grad students so that they can spend 20% of their time on blue sky research ideas. I get that time and then some without throwing poor kids under the bus. Without even asking, really. And make 3x what I would've made as a slave driver to boot.
Academia sucks, especially if you want to do good research. Seriously, outside of tippy-top elite institutions that can get the very best and brightest phd students, colleges and universities are some of the absolute worst places to do research these days.
And if you want to teach CS, don't bother with a PhD. Just ad junct a bit on the side of an industry gig, do not-insanely-bad, and you'll be full time within a year or two. If you want. Which you very likely wont, because teaching one class on the side while engineering is a lot more fun than teaching full time. Teaching is exhausting, even for people who love teaching.
Having started at a lower ranked school and ended at a higher ranked school, I can give you the following advice (which mostly applies to technical fields). First, do not fall into the trap of thinking that where you get your PhD, or whether you get your PhD, is some kind of indication of your worth. What matters is the quality of your work. People at top schools can do mediocre research, and people at low-ranked schools can change an entire field. Do the best research work you can do; nobody is going to care where the work was done.
Second, set expectations with your advisor -- you are not there to further your advisor's career, you are pursuing your own career goals and are not just cheap labor. If you start doing random tasks that your advisor needs done, tasks that are not related to your own research, you will find yourself doing those tasks all the time. It is like any other job, you need to make it clear that you are not going to work outside of your role.
Finally, and most important of all, keep your dignity and remember that you can always take a more satisfying job (for better pay). Make sure you pick up a master's degree ~2 years into the PhD program, it helps if you want to leave. You should never feel like you are trapped; years spent in a PhD program will count as years of relevant employment at most tech companies (at least those I am familiar with). The only reason you need a PhD is if you want a tenure track position, and you should know that junior faculty face a new lineup of BS as they work towards tenure.
Second, set expectations with your advisor -- you are not there to further your advisor's career, you are pursuing your own career goals and are not just cheap labor. If you start doing random tasks that your advisor needs done, tasks that are not related to your own research, you will find yourself doing those tasks all the time. It is like any other job, you need to make it clear that you are not going to work outside of your role.
Finally, and most important of all, keep your dignity and remember that you can always take a more satisfying job (for better pay). Make sure you pick up a master's degree ~2 years into the PhD program, it helps if you want to leave. You should never feel like you are trapped; years spent in a PhD program will count as years of relevant employment at most tech companies (at least those I am familiar with). The only reason you need a PhD is if you want a tenure track position, and you should know that junior faculty face a new lineup of BS as they work towards tenure.
We try to make “not attending top schools” not as important. We most often fail in doing that, of course, but the thought at least should be there.
It's over for you. Cry yourself to sleep.
Really? What should I do?
> What should I do?
Don't feed the trolls and instead read both of the replies to your original comment that focused on constructive advice.
Don't feed the trolls and instead read both of the replies to your original comment that focused on constructive advice.
Would you keep crying all your life on this "prestige" thing ? If you think you're better then prove it
Are you saying I'm inferior?
You said multiple times "others are perceived better". Don't go into semantics now. Admit it, you have. Brand NCSU < Brand Harvard, no matter how much you cry on HN.
That's rough dude. Why do you think I'm such a failure?
I'm not actually sure what semantics you're referring to, btw. AFAIK you went to CMU ECE (not SCS) and not Harvard so I'm not so sure you can make that comparison anyway. How exactly do you think I should prove it?
I'm not actually sure what semantics you're referring to, btw. AFAIK you went to CMU ECE (not SCS) and not Harvard so I'm not so sure you can make that comparison anyway. How exactly do you think I should prove it?
Have fun trolling or crying. Don't know which one is worse.
What part of what I said is trolling?
Yikes, jesus christ. I wonder what they think of people at "regular companies" like mine or people that graduated from "regular schools".
I myself graduated as one such commodity PhD student, and I don’t consider it as a derogatory term, mere reality. Contrast to regular jobs, the PhD nowadays is more like the arts where the top candidates disproportionately reap the benefits. If you’re not at a top tier university or a star advisor your academic prospects are middling from the beginning. Not saying you can’t make it happen, a lot of my friends do, but it’s a slog and the system is biased against you. In the end there’s limited money and you’re directly competing the stars I just talked about before. Some people are just monumentally better at everything so it’s just not fun competing with a very small pie at hand.
The farther you are away from the top schools, the less the faculty are likely to see you as a potential peer or someone who might inherit or carry on their legacy. Combined with control over funding this produces a dangerous power dynamic.
Lets put it in context:
>In fact, most of my mentors at Brain encouraged me to enroll in the PhD program. Only one researcher strongly discouraged me from pursuing a PhD, comparing the experience to “psychological torture.” I was so shocked by his dark warning that I didn’t ask any follow-up questions, he didn’t elaborate, and our meeting ended shortly afterwards.
What he did was discounted the unpopular and negative opinion because it was so polarising: “psychological torture”.
>In fact, most of my mentors at Brain encouraged me to enroll in the PhD program. Only one researcher strongly discouraged me from pursuing a PhD, comparing the experience to “psychological torture.” I was so shocked by his dark warning that I didn’t ask any follow-up questions, he didn’t elaborate, and our meeting ended shortly afterwards.
What he did was discounted the unpopular and negative opinion because it was so polarising: “psychological torture”.
> [Sundar Pichai] has emphasized that Google is an “AI-first” company
Maybe I need to lighten up here, but I find this "AI-first" branding to be really off-putting. What I fear it means is "We're going to add the world's biases and a bit more unpredictability into every single one of our features."
And in some ways it's like like a carpenter deciding to become a "wrench-first" operation. Shouldn't the first priority be the people affected by the things that you do, rather than a particular tool that you inflict upon them? And, great as it can be, is it really the right tool for every job? Maybe some of the problems the AI fairness team has exposed (or tried to) will push Google to realize that it need not be present in everything, let alone first.
Maybe I need to lighten up here, but I find this "AI-first" branding to be really off-putting. What I fear it means is "We're going to add the world's biases and a bit more unpredictability into every single one of our features."
And in some ways it's like like a carpenter deciding to become a "wrench-first" operation. Shouldn't the first priority be the people affected by the things that you do, rather than a particular tool that you inflict upon them? And, great as it can be, is it really the right tool for every job? Maybe some of the problems the AI fairness team has exposed (or tried to) will push Google to realize that it need not be present in everything, let alone first.
> I find this "AI-first" branding to be really off-putting
Honestly my first reaction was a snarky "Well, your moderation and support channels certainly are" given all the horror stories we hear about people getting locked out of their entire google accounts with no recourse or access to humans to appeal / investigate.
Honestly my first reaction was a snarky "Well, your moderation and support channels certainly are" given all the horror stories we hear about people getting locked out of their entire google accounts with no recourse or access to humans to appeal / investigate.
I kind of love how google has destroyed their reputation on the customer support side and doesn't seem to care. I believe at a certain point it will affect their bottom line. They must be betting that they develop passable AI support before that happens.
Did Google ever have a good reputation for customer support?
Your conclusion isn't wrong, but I think you are thinking about it the wrong way.
"AI-first" isn't because Sundar thinks that AI is the best solution for Google's problems. "AI-first" is because Sundar thinks that getting good at AI is an important long-term investment for Google.
So this is less like a carpenter deciding to use a wrench for every operation because they think that the results will be good and more like a carpenter deciding that they should find a project to use a wrench on because they want to learn how to use one.
"AI-first" isn't because Sundar thinks that AI is the best solution for Google's problems. "AI-first" is because Sundar thinks that getting good at AI is an important long-term investment for Google.
So this is less like a carpenter deciding to use a wrench for every operation because they think that the results will be good and more like a carpenter deciding that they should find a project to use a wrench on because they want to learn how to use one.
It reminds me of my managers talking about "focusing on moving to the cloud", without any real analysis whether it makes sense from technical or cost standpoint. It's purely hype driven.
Because it's the future and HBR (or some such "authority") told them so.
[deleted]
> For example, my coworkers let me take the lead in writing an academic paper about TensorFlow 2, even though my contributions to the technology were smaller than theirs.
Sounds like some folks didn't like writing! :)
Sounds like some folks didn't like writing! :)
The relevant question isn't who wrote the bulk of a paper - it's whose name appears first on the list of authors :).
This guy is [1], if I found the right paper. Though the order appears to be alphabetical.
[1] https://arxiv.org/abs/1903.01855
[1] https://arxiv.org/abs/1903.01855
It's not alphabetical.
Akshay A..., Akshay N..., Ale..., All..., Ash..., Asi..., I..., J..., M..., R..., S...
Looks alphabetical for me
Looks alphabetical for me
It literally says "Authors listed in alphabetical order." right in the paper.
I just don't get some HN assertions...
I just don't get some HN assertions...
> While I’m interested in machine learning, I’m not convinced that today’s AI is anywhere near as profound as electricity or fire
This seems a not too uncommon experience among CS people where machine learning loses its interest once they understand the magic
This seems a not too uncommon experience among CS people where machine learning loses its interest once they understand the magic
In 1850, few people felt that electricity was particularly profound either. That year, William Gladstone, Chancellor of the Exchequer, asked Michael Faraday why electricity was valuable. Faraday replied, "One day sir, you may tax it." [1].
The electric motor had been invented (by Faraday), but it couldn't do any useful work and had to sit in a pool of mercury to function. Electric lighting had been invented, but the filaments were dim and burned out too quickly to be practical. There was no way to generate or distribute electricity at scale. To most people, electricity looked like a toy (and a terrifying toy at that). It would take decades of invention and refinement to get to the point where electricity was useful, reliable, and ubiquitous. But the seeds for doing so were there in 1850.
So when Sundar Pichai says AI is more profound than electricity, it's possible he's right. It's just that we're still in its 1850, so we can't see it yet.
[1] https://en.wikipedia.org/wiki/Electricity#Cultural_perceptio...
The electric motor had been invented (by Faraday), but it couldn't do any useful work and had to sit in a pool of mercury to function. Electric lighting had been invented, but the filaments were dim and burned out too quickly to be practical. There was no way to generate or distribute electricity at scale. To most people, electricity looked like a toy (and a terrifying toy at that). It would take decades of invention and refinement to get to the point where electricity was useful, reliable, and ubiquitous. But the seeds for doing so were there in 1850.
So when Sundar Pichai says AI is more profound than electricity, it's possible he's right. It's just that we're still in its 1850, so we can't see it yet.
[1] https://en.wikipedia.org/wiki/Electricity#Cultural_perceptio...
So when Sundar Pichai says AI is more profound than electricity, it's possible he's right. It's just that we're still in its 1850, so we can't see it yet.
You can make a claim that "This might be revolutionary! It's just too early to tell." about literally anything.
You can make a claim that "This might be revolutionary! It's just too early to tell." about literally anything.
Yeah, but if we're looking for potential revolutions, we already have a bunch of data that points towards AI in the possibility space.
Just be aware that you're using selection bias to substantiate your point here. There were many more claims along the same lines that didn't pan out as well and didn't turn out as well. I even say this as someone who believes AI/ML will change the world (though I'm with MJ on this one).
I can use exact same analogy to prove that horseshit is next electricity. I don’t know why people do reasoning by analogies and why people accept analogies for reasoning.
There are almost seven times as many people alive today as there were in 1850 -- we may be in 1850, but the number of clever people driving advances is substantially larger.
They are also spread over a substantially larger spectrum of scientific disciplines.
There was no ML in 1850, and the word "tensor" was invented in 1846. No biochemistry, no genetics, no quantum mechanics.
There was no ML in 1850, and the word "tensor" was invented in 1846. No biochemistry, no genetics, no quantum mechanics.
I'm a big ML proponent, both because of work and personal interest. Saying it is like electricity is idiotic, and only makes false expectations about ML worse.
But I would not say that makes it uninteresting. Nor does it make it irrelevant from a business perspective. There are still lots of beneficial ways to use ML, and lots of different angles from which to derive intellectual satisfaction. I suppose if someone went into the field expecting I Robot type stuff, they might be disappointed, but it's hard to imagine someone with a CS background having such unrealistic expectations of what ML actually is.
But I would not say that makes it uninteresting. Nor does it make it irrelevant from a business perspective. There are still lots of beneficial ways to use ML, and lots of different angles from which to derive intellectual satisfaction. I suppose if someone went into the field expecting I Robot type stuff, they might be disappointed, but it's hard to imagine someone with a CS background having such unrealistic expectations of what ML actually is.
> This seems a not too uncommon experience among CS people where machine learning loses its interest once they understand the magic
Maybe. I went into my program with no preconceived ideas that there was 'magic' happening here. What I got out of it was a real curiosity about how humans think and how that may not be a magical as we think either.
Long-term I cannot see AI being anything but the paradigm-smashing artifact it's sold as. But progress is (relatively) slow on a human scale, so we find incremental progress ultimately lackluster.
If you take it through that lens it becomes more reasonable and you find yourself less skeptical that, just as we have no flying cars, we have no AGI or that all work has been replaced.
Maybe. I went into my program with no preconceived ideas that there was 'magic' happening here. What I got out of it was a real curiosity about how humans think and how that may not be a magical as we think either.
Long-term I cannot see AI being anything but the paradigm-smashing artifact it's sold as. But progress is (relatively) slow on a human scale, so we find incremental progress ultimately lackluster.
If you take it through that lens it becomes more reasonable and you find yourself less skeptical that, just as we have no flying cars, we have no AGI or that all work has been replaced.
Once you have a handle on the deep learning tools, it seems like it’s more like configuration and less like programming.
I didn't expect to see this here! Happy to answer questions. (There's also a discussion on Twitter: https://twitter.com/akshaykagrawal/status/136827691658779443...)
Read this on the weekend, and found a interesting branch of mathematics from your links. Thanks so much!
I am sure you need to stay politically correct but do you see places where Google Brain can do better or improve further?
> I suspect that convex optimization has the potential to become a powerful, widely-used technology.
I would say it has achieved that status already :)
Deep learning is getting more than its share of hype. Convex optimization is a jewel of computational science.
I would say it has achieved that status already :)
Deep learning is getting more than its share of hype. Convex optimization is a jewel of computational science.
This is incredibly well-written and a great peek behind the curtain of industrial vs academic ML research.
One question: can you elaborate a little on your point about intellectual diversity?
One question: can you elaborate a little on your point about intellectual diversity?
Sure! While Google Brain is focused specifically on AI, Stanford as a whole conducts research across a very wide range of disciplines. As a result, my network at Stanford has a more diverse collection of interests. As a PhD candidate I have worked with or am connected to people interested in neuroscience, radiation treatment planning, finance, control, economics, computer systems, and (yes) machine learning, among other topics.
Of course there's nothing wrong with focusing specifically on AI. It's just that I wanted to develop my research sensibilities in a more diverse environment.
edit: & thank you for the kind words!
Of course there's nothing wrong with focusing specifically on AI. It's just that I wanted to develop my research sensibilities in a more diverse environment.
edit: & thank you for the kind words!
how much real-world collaboration do you get to do with students, postdocs and faculty in those other disciplines? (ie; what is the experience like when one does a phd in computational tools as opposed to a specific stem domain?)
I think it depends on many things, such as whether people at your university are generally collaborative, and whether your advisor or you are well connected. Most of my time is spent on tools or fundamental research, since that's my primary interest; but sometimes people will reach out to me (or vice versa) about starting a collaboration. But even just exchanging notes and chatting with researchers in other disciplines has been fun.
But that is just my experience. Some of my labmates had PhD's in which the majority of their work was collaborative, across disciplines.
But that is just my experience. Some of my labmates had PhD's in which the majority of their work was collaborative, across disciplines.
I have nothing of value to add except that seeing someone do what they love and enjoy, greatly warms my heart.
Good luck with everything Akshay!!
Good luck with everything Akshay!!
I always get so excited when reading about Brain, X, Parc, etc... But sadly I just don’t have the academic background to get into one of those places :(
Really? If anything it makes me really angry. I generally despise seeing people happy, especially people that have gotten everything in life (as someone that has definitely not).
What an interesting way to view the world...
What an interesting way to view the world...
I once did Convex Optimization course with Stephen Boyd and it was totally amazing! I think it is still available here: https://www.edx.org/course/convex-optimization and the e-book https://web.stanford.edu/~boyd/cvxbook/ is also freely available. I am quite jealous. :) Good luck with your PhD!
This course was great
Thank you for the writeup and don't be put off by all the snarky comments here :)
Fantastic blog post! Thanks for sharing.
I always find fairly positive accounts of graduate school to be quite interesting. Without a doubt, you can always find someone in the comments (if it's something posted in a format such as this) with a chip on their shoulder from their own graduate school experience (or lack thereof). Also I find it interesting to think about places similar in impact to Xerox PARC, which people usually admire after everything unfolds and becomes a part of history. A good chunk of people from today would probably avoid a place like PARC back then in order to opt for a more stable or known job.
I always find fairly positive accounts of graduate school to be quite interesting. Without a doubt, you can always find someone in the comments (if it's something posted in a format such as this) with a chip on their shoulder from their own graduate school experience (or lack thereof). Also I find it interesting to think about places similar in impact to Xerox PARC, which people usually admire after everything unfolds and becomes a part of history. A good chunk of people from today would probably avoid a place like PARC back then in order to opt for a more stable or known job.
[deleted]
> I’m not convinced that today’s AI is anywhere near as profound as electricity or fire
@akshayka can you elaborate a bit more on why you think that
@akshayka can you elaborate a bit more on why you think that
The answer is simple: electricity is a prerequisite to artificial intelligence, and fire was necessary to provide us with the nutrition we needed to harness the power of electricity.
Modern ML certainly has some cool applications, but none seem revolutionary. Can you list any that are?
Off the top of my head:
When paired with a robotic medium, we'll be able to automate most pattern driven jobs like construction, welding, surgery, anything really that doesn't require human element to it and is more of a pattern matching behavior.
This is extremely revolutionary and will put a ton of people out of work. Any pattern-dependent job that does not require human interaction will be at risk, even coding (as we've seen by GPT-3's ability to generate code). Lawyers, sales people, cops, these jobs will be ok, but there will be less demand for humongous medical teams, software teams, etc. Bc all the entry-level work can be handled by AI, and just need some QA from a few engineers.
When paired with a robotic medium, we'll be able to automate most pattern driven jobs like construction, welding, surgery, anything really that doesn't require human element to it and is more of a pattern matching behavior.
This is extremely revolutionary and will put a ton of people out of work. Any pattern-dependent job that does not require human interaction will be at risk, even coding (as we've seen by GPT-3's ability to generate code). Lawyers, sales people, cops, these jobs will be ok, but there will be less demand for humongous medical teams, software teams, etc. Bc all the entry-level work can be handled by AI, and just need some QA from a few engineers.
This is a wildly optimistic take on what AI can accomplish, and a misunderstanding of the difficulties in all those tasks. If AI can do the first set of jobs, it will be able to replace sales people, cops, and lawyers too. But there's nothing, yet, to indicate that there isn't a wide gulf between the current state of the art and the kind of generalized intelligence you're talking about.
I mean it can already replace certain tasks done by sales people, cops and lawyers.
Three lawyers with the right set of tools (e.g. for discovery, writing legal briefs, etc.) can be more productive than twenty lawyers.
Three lawyers with the right set of tools (e.g. for discovery, writing legal briefs, etc.) can be more productive than twenty lawyers.
Yes but for jobs that require emotion like sales, the current Neural Network paradigm is just not gonna cut it. I am never buying from a robot, it’s missing an element of the mind that we still haven’t even categorized the biological side of, even if it sounds like a “real person”.
The state of the art of today’s DNN, I agree, is far away. But the peak of today’s architecture is yes, replacing construction, welding, and surgery. Those actually might happen quickly.
The state of the art of today’s DNN, I agree, is far away. But the peak of today’s architecture is yes, replacing construction, welding, and surgery. Those actually might happen quickly.
Besides, the more imminent threat to "our jobs" is not artificial intelligence but the brutal combination of globalization and the increasingly effortless ability to work remotely. If it doesn't matter anymore where the person behind Slack or Zoom is located, then why should they be in the same country. Before AI catches up to our jobs, the jobs will already have been taken away from us by the forces of globalization.
That would be revolutionary! Like others in the comments, I haven't seen evidence that this can be done with today's technology. But I would be happily be proved wrong, and would love to learn more.
All that from today's AI? The best I've seen from today's AI is a few gpt-3 articles that are clearly in the uncanny valley and a car which may or may not mistake an exit ramp wall for a passing lane.
When you demo some tech to 5 year old and he mistakens it for magic, I think it’s revolutionary enough. In current state of art, Atari RL, AlphaGo, AlphaFold, GPT3, GANs, Pix2pix, CLIP, DeepFakes etc are very very revolutionary. You can take any of these and go back in time even just 15 years ago, people won’t believe what was accomplished.
However, I do believe we are far and away from generalized intelligence.
However, I do believe we are far and away from generalized intelligence.
Yes, those applications are awesome. I just think they don't hold a candle to (sorry for the pun) fire, or electricity. But maybe that's not a useful comparison, since very few people likely think (today's) AI to be _that_ revolutionary.
The comparison to fire and electricity aside --- you're absolutely right, we can do very interesting things with machine learning & automatic differentiation these days!
The comparison to fire and electricity aside --- you're absolutely right, we can do very interesting things with machine learning & automatic differentiation these days!
Protein folding seems like it could be a revolutionary application.
I'm always a little skeptical of someone who gets all their degrees from the same university, even it it is Stanford. They aren't exposed to as diverse of a group of thoughts and ideas as someone who goes else where and proves themselves to other people with differing opinions.
Of particular note:
"Google’s CEO Sundar Pichai (who believes AI is “more profound than electricity or fire”) has emphasized that Google is an “AI-first” company, with the company seeking to implement machine learning in nearly everything do"
Nearly everything they do with your data.
"Google’s CEO Sundar Pichai (who believes AI is “more profound than electricity or fire”) has emphasized that Google is an “AI-first” company, with the company seeking to implement machine learning in nearly everything do"
Nearly everything they do with your data.
Not sure if a part of your post got cut off? As framed, it seems rather inflammatory without much direction or practical addition to the conversation. Perhaps I'm missing something.
Nope, nothing got cut. My post stands as I wrote it. It is not ad-hominem and not what-aboutery. It is very much a reasonable question to ask what they use their much-touted AI on and how this relates to the well documented privacy concerns around Google.
The original is actually more correct it seems.
IMO they are clearly betting (parts of) the farm on this as can be seen by the (often) ridiculous results many of us have witnessed in both search results quality and ad matching quality.
IMO they are clearly betting (parts of) the farm on this as can be seen by the (often) ridiculous results many of us have witnessed in both search results quality and ad matching quality.
But ignoring that one persons warning about PhD still sounded like he had the typical arrogance contained by every person who ever did a PhD (including me). If someone says “don’t do it” and your reaction is to not even follow up with questions perhaps you should acknowledge you already know why they’re saying that! It’s still not clear to me a person of this cadre need a PhD in a technical field in this day and age; they might contribute just as much faster if they chose any other route.