Jeff Dean interview: Machine learning trends in 2020(venturebeat.com)

174 pointsby panabee7 anni fa57 comments

Jeff Dean interview: Machine learning trends in 2020

https://venturebeat.com/2019/12/13/google-ai-chief-jeff-dean-interview-machine-learning-trends-in-2020/

59 comments

The 2020 trends question is right at the end. Summary:

- much more multitask learning and multimodal learning

- more interesting on-device models — or sort of consumer devices, like phones or whatever — to work more effectively.

- AI-related principles-related work is going to be important.

- ML for chip design

- ML in robots

rurban7 anni fa

This the first nonsense talk by Jeff Dean. AI doesn't help at all it battling climate change, only politics do help. The models are accurate enough for centuries, AI would only help in hard forecasting in the usual 2 weeks window on local events. Long term on global scale there's no AI needed at all. So it looks like he ran out of topics to entertain himself. Or he went politician. Which would be a welcoming change.

cyorir7 anni fa

I don't think AI has to be entirely useless in battling climate change. Dean has made other comments on AI and climate change:

https://venturebeat.com/2019/12/16/ai-experts-urge-machine-l...

"Dean said the company is thinking of including more information in Google search results to give users a predicted carbon output for choices they make, like ordering a certain product."

An AI system that predicts carbon output for consumer choices and delivers that to consumers could be useful in battling climate change, even if only in a small way.

thundergolfer7 anni fa

I’d agree that Jeff there is a man with only a hammer walking around telling us that Climate Change is a nail. Engineers can’t fix everything.

Of course the engineering is important, but politics is 1000% the road block on climate action and that is where efforts should be focused,

natalyarostova7 anni fa

> that is where efforts should be focused

Are all humans fungible? You don't think it's valuable to have some errant geniuses trying to think of weird and creative ways to solve climate change through unconvential engineering? It's not as though they'd be good politicians.

thundergolfer7 anni fa

Of course yes, that's valuable. But the political focus shouldn't be lost in the messaging. Google needs to take a political stand on Climate Change, first and foremost. The engineering investment is secondary. We already have the technology to solve our climate problems. We don't need the fancy AI solutions. We just need to implement what we have already right now.

That needs to be made abundantly clear to everybody.

If Google is asked "what are you doing to address Climate Change?" and we hear predominantly engineering talk, then they're being disingenuous.

panarky7 anni fa

Perhaps the Singularity can figure out how to clean up the political mess we've made.

summerlight7 anni fa

Did you go the talk or have the slide/video? It looks like you know the details and I'm interested in this but cannot find its details yet.

Nashooo7 anni fa

Does anyone have tips on how a European based developer with machine learning expertise can get involved with projects battling climate change like Jeff is talking about here?

listentojohan7 anni fa

There's Tomorrow - and they're hiring. https://www.tmrow.com/

mrischard7 anni fa

We're doing some interesting work in this field at Cervest (based in London). https://www.cervest.earth/

chaosphere21127 anni fa

There are a variety of European climate agencies; depending on what country you live in / are interested in being in, there's:

- UK: Met Office https://www.metoffice.gov.uk/about-us/careers/vacancies)

- France: Institut Pierre Simon Laplace https://www.ipsl.fr/en/ (their job page is busted)

- Germany: Max Planck Institute https://www.mpimet.mpg.de/en/institute/opportunities/

- Netherlands: KNMI https://www.werkenvoornederland.nl/organisaties/ministerie-v...

I'm sure there are more, but these are the ones that I've worked with in the past. European climate research agencies seemed to have their act together a lot better than US ones; before I left the industry, they were much further on the path of moving computation to the commercial cloud (instead of trying to keep up in the supercomputer arms race, which always seemed like a losing proposition to me), and I think ML has been increasingly integrated into climate models as a way to approximate complex dynamical systems.

In terms of which of these labs had their head screwed on straight from a technical side, it was probably KNMI / Met Office, followed by Max Planck and then IPSL.

kot-behemoth7 anni fa

My current client is a sophisticated AI/ML startup/consultancy, Faculty (https://faculty.ai). They have extensive experience in a variety of areas, and do some cutting-edge stuff. If you're intested, either ping me (email in the profile) and I can connect, or use the website.

ace322297 anni fa

+1 for Faculty - very smart team

ViralBShah7 anni fa

I am happy to share my perspective on this. I feel a really good blog on this topic is Bret Victor's blog post - What can a technologist do about climate change?

http://worrydream.com/ClimateChange/

One of the clear recommendations is - contribute to Julia. A lot of what Bret said in 2015 has actually panned out. Julia has become a powerful language for scientific computing and machine learning. As a result it is being used in climate projects such as Climate Machine (MIT and Caltech). The Julia Lab at MIT participates in this project:

Climate Machine: https://clima.caltech.edu/ Github: https://github.com/climate-machine/ Julia Lab: http://julia.mit.edu/

Contributions to Julia packages (compiler, stdlibs, math packages, ML packages, parallel computing) will end up finding their way into climate research, because of the extensive reuse of code within the Julia ecosystem. Specifically capabilities such as Zygote.jl (https://github.com/FluxML/Zygote.jl) for differentiable programming have the potential to dramatically make it easy to apply ML techniques to scientific codebases. Compiler contributors are hard to come along, so all contributions to compiler technology are incredibly valuable. The DiffEqFlux.jl ecosystem in Julia is a good example of combining mechanistic models with ML (https://github.com/JuliaDiffEq/DiffEqFlux.jl). Hop on to the Julia slack channel or discourse to dig in deeper.

Another thing Bret Victor speaks about in his blog post is working with agencies such as ARPA-E on advanced projects. Julia Computing is participating in an ARPA-E project to bring these capabilities to many energy related simulation and development technologies. This press release gives a broad idea:

ARPA-E press release: https://www.energy.gov/articles/department-energy-announces-... Funded projects: https://arpa-e.energy.gov/sites/default/files/documents/file... Julia Computing press release: https://juliacomputing.com/communication/2019/12/09/arpa-e.h...

m00dy7 anni fa

Hi,

I'm also working something similar to this. Would you leave your contact info ?

ma2rten7 anni fa

Google Brain has several offices in Europe.

summerlight7 anni fa

https://arxiv.org/abs/1906.05433

This paper seems relevant to the climate change part. This is more about what to do rather than technologies.

m0zg7 anni fa

I hope one of those trends will be fixing (or abandoning) TensorFlow, because it's really disorganized, full of legacy and mysterious behaviors, and in some places just plain badly engineered.

mstokholm7 anni fa

Is this entire talk up on YouTube somewhere?

narrator7 anni fa

Somebody tell Jeff about Jevon's Paradox, or actually don't bother.

deepaksurti7 anni fa

Jevon's Paradox [1]

An example from [1]:

```

Jevons observed that England's consumption of coal soared after James Watt introduced the Watt steam engine, which greatly improved the efficiency of the coal-fired steam engine from Thomas Newcomen's earlier design. Watt's innovations made coal a more cost-effective power source, leading to the increased use of the steam engine in a wide range of industries. This in turn increased total coal consumption, even as the amount of coal required for any particular application fell. Jevons argued that improvements in fuel efficiency tend to increase (rather than decrease) fuel use ...

```

[1] https://en.wikipedia.org/wiki/Jevons_paradox

whack7 anni fa

Jevon's paradox only occurs under very specific circumstances. I would bet on it not happening in the context Jeff Dean is referring to.

https://en.wikipedia.org/wiki/Jevons_paradox#Cause

> The size of the direct rebound effect is dependent on the price elasticity of demand for the good.[11] In a perfectly competitive market where fuel is the sole input used, if the price of fuel remains constant but efficiency is doubled, the effective price of travel would be halved (twice as much travel can be purchased). If in response, the amount of travel purchased more than doubles (i.e. demand is price elastic), then fuel consumption would increase, and the Jevons paradox would occur. If demand is price inelastic, the amount of travel purchased would less than double, and fuel consumption would decrease. However, goods and services generally use more than one type of input (e.g. fuel, labour, machinery), and other factors besides input cost may also affect price. These factors tend to reduce the rebound effect, making the Jevons paradox less likely to occur.

wallace_f7 anni fa

Here, I know your meaning is about market adoption of AI, but also on the topic of AI this made me wonder why it intuitively feels to me like Jevon's model would be inadequate, but do I only have this intuition because of my education? I think there's another way: critically test the statement:

>Jevons argued that improvements in fuel efficiency tend to increase (rather than decrease) fuel use

In other words, I can ask: does aggregate fuel usage depend on fuel efficiency? If I state the question this way and ask myself to answer it, it starts to become obviously an incomplete model to use.

This may seem so pedestrian but it is so confusing for me to think about in terms of conceptualizing what structure was needed to recognize a poor prediction model and considering a better one. This seems so different than what AI is today.

roenxi7 anni fa

I don't quite follow your comment but that isn't about to stop me guessing what you are asking and providing the intuitive model for the paradox.

The bad intuition is if X units of a resource are being used to achieve an outcome then providing the option of getting the same outcome with (X - something) will result in less resource use.

However, that is ignoring the economic principles of supply and demand to only focus on current use and demand. Reframing it from the supply side: it used to make sense to supply X units to do so much. Now supplying X units can do even more than so much.

So the economics of the situation are unlikely to cause a reduction in supply, because if it made sense to supply X units of resource before, it really makes economic sense to supply it now. In fact, since the resource is now more useful (efficiency rose) it probably makes sense to supply even more of it.

If Jevons's Paradox actually appears paradoxical, the root cause is a misunderstanding of supply and demand. If efficiency is modelled by moving the supply/demand curves around on a supply/demand chart it is pretty obvious what is going on; efficiency gains are equivalent to moving the supply curve under that model.

paganel7 anni fa

He probably does know about it because he seems he’s a well-read guy, but I think is too late for him and for people like him now: the pay they receive is too good for them to leave it all for some “principles” and on top of that I think the’ve also managed to acquire come cognitive dissonance traits that allow them to get out of bed in the morning and go to work without feeling guilty.

Otherwise I cannot understand how he can really think that “more AI” is going to help with deforestation, as in “more AI” probably means less overall costs for bad people in the Amazon (where your major costs are people-related, it’s phisically very demanding cutting down trees in an Equatorial climate) which in turn means more trees being cut down. And this is just the beginning of it.

nl7 anni fa

Jeff is right.

Training a machine learning model for any common task is getting radically cheaper and more energy efficient.

This is a combination of both better hardware (eg, Google TPUs) and better optimisation of training techniques (which is mostly done outside Google).

Eg, DAWN Bench[1] benchmarks training of ReseNet50 to 93% accuracy. This used to take days.

Now the FastAI group has shown you can do it in 18 minutes on 16 p3.16xlarge on AWS Spot instances. This is a huge energy saving.

Huawei has shown you can do it on 16x8xV100s on their cloud in less than 3 minutes.

[1] https://dawn.cs.stanford.edu/benchmark/

choppaface7 anni fa

Can’t wait for Cerebras. They can probably do an ImageNet in a couple seconds.

nl7 anni fa

I'll believe it when I see independent tests.

paganel7 anni fa

“Energy saving” does not mean “cool, then we won’t use/consume that saved energy anymore”, it means “cool, now there’s more stuff for us to do/build with the same energy as before”. And more stuff to do/build means more pressure on the environment.

aguyfromnb7 anni fa

Electronic computers have become vastly more energy efficient since they became widely used in the 60s. On net, do we spend more or less energy on computers now than we did then?

That's the paradox. The cheaper energy becomes, the more uses we find for it.

mkolodny7 anni fa

From the article:

> all the stuff we trained in our Google Data Center — the carbon footprint is zero. Because … basically, all of our energy usage comes from renewable sources.

outside12347 anni fa

... which takes all of those renewable sources away from the rest of our energy consumption, so net net, their data centers cause more carbon.

dwaltrip7 anni fa

It's actually most likely the reverse. The more renewables are manufactured and used, they more they will be improved and made increasingly cost-effective, which will in turn make them more widely useful and competitive. It's a positive feedback loop.

This is what we have been seeing the past 10+ years, and of the main reasons why renewables are starting to be directly competitive with fossil fuels in some regions, even without subsidies. This is a huge turning point that we are on the cusp of.

rejschaap7 anni fa

Is Jevons paradox really a problem when your carbon footprint is zero?

> VentureBeat: One of the things that’s come up a lot lately, you know, in the question of climate change — I was talking with Intel AI general manager Naveen Rao recently and he mentioned this idea [that] compute-per-watt should become a standard benchmark, for example, and some of the organizers here are talking about the notion of people being required to share the carbon footprint of the model that they trained for submissions here.

> Dean: Yeah, we’d be thrilled with that because all the stuff we trained in our Google Data Center — the carbon footprint is zero.

dredmorbius7 anni fa

The Jevons Paradox describes not just individual responses to lower prices, but the overall market response. Which at its largest expanse is total global demand.

By way of analogy, the total industrialised world as of 1900 was mostly the US, UK, France, and Germany, with a few odd other millions from Canada, Australia, and bits of Europe.

Not only did those 200 million or so souls see a tremendous personal increase in services (and resources) consumed, but:

- The population of industrialised nations swelled by a factor of about 5, to 1 billion (the G7 + EU).

- Another two billions grew and increased economic activity in China and India.

- The Rest of the World grew to another 5 billions.

The G7 per capita GDP is about $40k. Depending on what figures you look at, China lands somewhere between about $8k - $14k/person. The rest of the world (including India) is lucky to hit an average of $5k, or about 1/8th the G7, whilst being 5 times its population.

Put another way, raising the RoTW to G7 standards would swallow 40x current global economic activity, and that's presuming the G7 + EU hold still, something few economic models allow for.

Or: Jevons has a lot of mileage yet.

Regards Google's "zero carbon" datacentres: if you don't account for the rest of the market's response, this simply lowers costs for other entrants to make claims which Google have effectively abandoned, absent some means of otherwise increasing prices. Fighting resource utilisation by increasing efficiency alone is like fighting for peace or fcking for virginity. It tends to self defeat.

To decrease overall* utilisation, you've got to increase costs or decrease demand. Taxes, sink utlisation charges (e.g., carbon cap-and-trade), or some form of suasion against resource utilisation, might work. Efficiency itself simply lowers costs, which is to say, increases the supply function, and moves total quantity consumed up.

account734667 anni fa

>> all the stuff we trained in our Google Data Center — the carbon footprint is zero

That is a lie unless Jeff Dean lives under different physics laws.

joatmon-snoo7 anni fa

It's a mix of (1) great work around sustainable processes and (2) nuances in defining carbon footprint.

(disclaimer: work at G)

https://storage.googleapis.com/gweb-sustainability.appspot.c...

https://www.google.com/about/datacenters/renewable/

account734667 anni fa

Buying renewable energy does not make your footprint zero. You can twist the definition until it suits your needs but if you properly count footprint of the chain, it cannot be zero.

asdfasgasdgasdg7 anni fa

Google buys high quality carbon offsets, not just renewable energy. It's not clear whether that makes the footprint zero. It surely depends on how you count. But their approach is less naive than "just buy RECs."

m0zg7 anni fa

Yeah, unless this is powered entirely by nuclear or renewables, or unless the emitted carbon is then extracted from the atmosphere, the carbon footprint of all this is most definitely not zero. It's like all those celebs buying carbon "offsets" after flying private to climate summits of all sorts. Dude, your carbon footprint is way, way, above zero, and you paying for "offsets" doesn't reduce it one iota.

wnkrshm7 anni fa

Even better that the offset projects aren't necessarily helping. Apparently it isn't even clear whether just planting trees anywhere has a net positive influence on combatting the greenhouse effect [0]. Trees may be required to be planted at suitable locations or else they actually contribute to warming.

[0] https://www.nature.com/articles/d41586-019-00122-z

buboard7 anni fa

even if the carbon footprint is zero (which it isnt) , all this energy is basically a large scale heater.

[deleted]7 anni fa

derangedHorse7 anni fa

I think google as a whole understands this concept. Isn't that why they've held off on the release of their self-driving car? I can't remember where I read this from but I remember an interview where someone said they didn't want to release their self-driving car to the world until it was X-times better than the average human driver, and extremely competitive with its price.

outside12347 anni fa

Why is there never a Google talk entitled "Building profiles on people with Machine Learning?"

option7 anni fa

Because you don’t disclose your core business practices at conferences

bytematic7 anni fa

They probably don't need machine learning since they have a more complete profile on us than we could personally create by just query haha

symplee7 anni fa

Unrelated, but for those who have never seen the Jeff Dean facts, behold: https://www.quora.com/What-are-all-the-Jeff-Dean-facts

Some highlights include:

-Jeff Dean's PIN is the last 4 digits of pi.

-He once shifted a bit so hard it ended up on another computer.

-He wrote an O(n^2) algorithm once. It was for the Traveling Salesman Problem.

-Jeff Dean once implemented a web server in a single printf() call. Other engineers added thousands of lines of explanatory comments but still don't understand exactly how it works. Today that program is known as GWS.

-There is no 'Ctrl' key on Jeff Dean's keyboard. Jeff Dean is always in control.

-Jeff Dean's watch displays seconds since January 1st, 1970. He is never late.

-Jeff's code is so fast the assembly code needs three HALT opcodes to stop it.

Keloo7 anni fa

- After Graham Bell invented the phone he had a missed call from Jeff Dean

majewsky7 anni fa

> Jeff Dean's watch displays seconds since January 1st, 1970.

At a Hacker Jeopardy some time ago (I think it was the one at 29C3), the final round ended in a tie, so a tie-breaker was needed. The tie-breaker question was "What is the current Unix timestamp?" The contestants struggled hard, leading the moderator to exclaim "For god's sake, don't you ever check the clock!?"

boris7 anni fa

- Jeff Dean once caused an AI winter so cold, even humans lost their intelligence.

GrryDucape7 anni fa

I don't like the idea of a computer that can think for itself, I don't like the idea of computers will replace humans jobs, I don't like the way we are heading.

cookie_monsta7 anni fa

I wouldn't like the way it was heading either if Netflix could manage to recommend a movie that I might actually like. As is, I'm not that worried.

a_imho7 anni fa

Hype does not care about whether it works as advertised on the tin. As long as it generates money you will be served buzzword whether you like it or not and you will pay for it. If lucky, the setback will only be financial.

2sk217 anni fa

You have nothing to worry about :-) I would recommend that you read the book by Gary Marcus and Ernie Davis for a country viewpoint.

lonelappde7 anni fa

Say hello to your rickshaw carrier for me, since you don't use cars, buses, or trains.

rimliu7 anni fa

Don't worry, in this case we are running in circles.

AllasAskBar7 anni fa