Google announces a new generation for its TPU machine learning hardware(techcrunch.com)
techcrunch.com
Google announces a new generation for its TPU machine learning hardware
https://techcrunch.com/2018/05/08/google-announces-a-new-generation-for-its-tpu-machine-learning-hardware/
116 comments
Instead of being able to buy your own nvidia gpu and run any cuda or opencl on it, now you can run only tensor flow on a tpu only in Google cloud.
How fantastic.
How fantastic.
Choice in the marketplace is still choice. And, Google might eventually decide to sell this to outside companies. Eventually the desire to build competitive advantages for Google Cloud might be surpassed by the desire to compete with NVIDIA. Commoditizing the complement is a powerful strategy.
Yeah seems like a way more abusive position to me frankly, a Google monopoly would be way worse than a NVIDIA one.
Not a Google monopoly but instead another choice. To push Nvidia and others.
And only using Tensorflow.
It is about choice and driving down cost. Love google keeps pushing the envelope which is good for everyone
We're working to provide more choice with PlaidML as well. You can use it to run Keras, ONNX, and TF models on AMD, Intel, and mobile/embedded GPUs, on most operating systems and OpenCL, LLVM, OpenGL, and soon CUDA and Metal, etc. We would welcome contributions for TensorFlow integration, ROCm support, and anything else that widens support to more people.
https://github.com/plaidml/plaidml
https://github.com/plaidml/plaidml
Cool! Are you considering Vulkan?
A couple people internally have looked at Vulkan, it seems like it should work really well but we're not actively working on it. You can get an idea of how Vulkan support would look by looking at the existing HALs:
https://github.com/plaidml/plaidml/tree/master/tile/hal
It's kind of cool to look anyway because it's a lot of value in less code than you might expect (complexity is in the layers above HAL).
If someone wanted to add support for Vulkan or something more ambitious like Qualcomm Hexagon or the RPi with QPU that would be pretty cool though.
https://github.com/plaidml/plaidml/tree/master/tile/hal
It's kind of cool to look anyway because it's a lot of value in less code than you might expect (complexity is in the layers above HAL).
If someone wanted to add support for Vulkan or something more ambitious like Qualcomm Hexagon or the RPi with QPU that would be pretty cool though.
TPUs seem like great hardware. Unfortunately google's insistence on keeping them in a walled garden is a major deterrent to building something that relies on their platform.
What makes sense. Would guess each cloud provider will do their own.
You seem to be missing the point again and again every time this topic comes up. There is a reason why TensorFlow is generic enough to run on many types of hardware. That allows people to port their code around. But there exists a world outside of tensorflow that you might not be aware of, and for that world this chip is not a competitor. That world is significantly larger than the TF world.
And that helps me get one in my own servers how?
Common library above is how. Top one today is TF.
So how does that get me a TPU in my own servers?
Does not make much sense to buy your own chips any longer.
TF removes the silicon from the equation. So if Nvidia cheaper use but if TPUs cheaper use. If something new that is cheaper then move. TF is the key.
TF removes the silicon from the equation. So if Nvidia cheaper use but if TPUs cheaper use. If something new that is cheaper then move. TF is the key.
Well until that actually happens I will be buying more NVIDIA stock.
I have done deep learning also and I don't see NVIDIA being an incumbent that is easy to be knocked out. I think we would need some distruption such as a new technique that is more performant or different design goals. Intel is still leading the way and I don't see NVIDIA being unseated with its head start anytime soon.
I have done deep learning also and I don't see NVIDIA being an incumbent that is easy to be knocked out. I think we would need some distruption such as a new technique that is more performant or different design goals. Intel is still leading the way and I don't see NVIDIA being unseated with its head start anytime soon.
I'm not as pessimistic. 3dfx and Matrox didn't look like easy to knock out opponents in the 90s and yet here we are. Markets are powerful incentive mechanisms.
"Slowly grinds the mill of the gods, but it grinds fine."
"Slowly grinds the mill of the gods, but it grinds fine."
Costs to compete in the silicon business have inflated massively since that era, its perhaps 100 or even 1000 times more expensive to build stuff like fabs (although NVIDIA dosent own fabs), and design chips.
But this makes it so Google is not dependent on Nvidia. But also push Nvdia.
>3dfx and Matrox didn't look like easy to knock out opponents in the 90s and yet here we are.
Matrox didn't give much of a damn to 3D gaming, 3Dfx were bust doing all the wrong things. It was certainly not "easy", but don't seems hard or impossible either at the time.
Matrox didn't give much of a damn to 3D gaming, 3Dfx were bust doing all the wrong things. It was certainly not "easy", but don't seems hard or impossible either at the time.
I will also change my Cable provider to something other than Comcast because I hate their service. Oh wait they are the only provider in my area...
An up-and-coming project in an up-and-coming language:
ArrayMancer: https://mratsim.github.io/Arraymancer/
Written in Nim, which looks and feels a lot like Python, but runs a lot like C++ or Rust, especially in the speed and FFI departments, and deploys like Go as a small single fast executable. Already supports CPU, CUDA and OpenCL. Still early, but with a lot of promise. Use is modeled on some mixture of PyTorch and Keras.
ArrayMancer: https://mratsim.github.io/Arraymancer/
Written in Nim, which looks and feels a lot like Python, but runs a lot like C++ or Rust, especially in the speed and FFI departments, and deploys like Go as a small single fast executable. Already supports CPU, CUDA and OpenCL. Still early, but with a lot of promise. Use is modeled on some mixture of PyTorch and Keras.
Self-plug: Gorgonia[0] can support OpenCL as well (but time and effort is required to write support libs). Would appreciate contributions. I recently gave a talk on how to interface with foreign libs[1]. Writeup soon to come.
[0]: https://github.com/gorgonia
[1]: https://www.youtube.com/watch?v=7R39CMW4MJQ
[0]: https://github.com/gorgonia
[1]: https://www.youtube.com/watch?v=7R39CMW4MJQ
I think NVidia was milking the cow in anticipation of moves from Google, AMD and MS in their space, leading to a pricing war.
Water cooling on their TPU is probably because they cost way more to buy than the extra cost of special cooling. It is also probably to increase “spec” on their new version without the best improvements in hardware/silicon.
> Google CEO Sundar Pichai said the new TPU is eight times more powerful than last year
Are we sure about this? He specifically said that a "pod" would be 8 times faster than last year, not the TPU itself.
And the picture in the background showed what looked like to be 8 racks of 64 TPUs (or maybe 32?). Until now a "pod" was a single rack of 64 TPUs. So if the new definition for a "pod" is 8 times as many TPUs as it was before, the result is less impressive...
Is there any actual spec released?
Are we sure about this? He specifically said that a "pod" would be 8 times faster than last year, not the TPU itself.
And the picture in the background showed what looked like to be 8 racks of 64 TPUs (or maybe 32?). Until now a "pod" was a single rack of 64 TPUs. So if the new definition for a "pod" is 8 times as many TPUs as it was before, the result is less impressive...
Is there any actual spec released?
Google started playing this game with the tpuv2, wherein they define a "TPU" as whatever they want to make it sound suitably impressive (I.e. Cloud TPU is four chips). This in turn led to nvidia calling the DGX-2 "the world's largest GPU
What does it matter? What we care about is the cost. The TPU gen 2 are about half as much as using Nvidia in the Amazon cloud for the same work. To me that is what matters. If the TPU 3 further drops the cost fantastic.
How many chips sounds a lot more like a pissing contest. What if it was a giant chip with actually a bunch of chips inside? Who cares?
How many chips sounds a lot more like a pissing contest. What if it was a giant chip with actually a bunch of chips inside? Who cares?
I care because assuming the cost of a single TPUv3 is roughly similar to the cost of a single TPUv2 (at launch), then paying for a "pod" of 256 or 512 (whatever a pod is this year) will be nowhere near the cost of 64 TPUv2 "pod".
So I care because I want to know if I will actually see this 8x speed improvement or if this is just marketing BS.
If NVIDIA tells me a 2080 is 8x time faster than a 1080, I know that a 2080 when released will be roughly the same price as the 1080 when it was released, and so I can expect to actually see a 8x cost/perf ratio improvement (if we put aside the fact that this are always best case scenarios)
So I care because I want to know if I will actually see this 8x speed improvement or if this is just marketing BS.
If NVIDIA tells me a 2080 is 8x time faster than a 1080, I know that a 2080 when released will be roughly the same price as the 1080 when it was released, and so I can expect to actually see a 8x cost/perf ratio improvement (if we put aside the fact that this are always best case scenarios)
You can already hook together TPUs. So, IMO, it is here or there what is happening in the data center. What I care about is how much is the cost. Time to train is more of a function of how much resources I want to use.
So what? I just want to know if the 8x speed increase was achieve at roughly the same cost (i.e same numbers of TPUs) or not.
If the 8x speed increase was achieve with 4x more TPUs (and roughly 4x the cost) like it seems it was, then this is just marketing bullshit, and we should really expect a 2x improvement in cost/perf ratio not 8x.
If the 8x speed increase was achieve with 4x more TPUs (and roughly 4x the cost) like it seems it was, then this is just marketing bullshit, and we should really expect a 2x improvement in cost/perf ratio not 8x.
I am also curious. But really it comes down to cost. Right now we see about 1/2 the cost of using TPUs versus Nvidia on the AWS cloud.
Will be interesting to see if the difference grows further with the TPU 3.0 or Google will just take larger margins.
Will be interesting to see if the difference grows further with the TPU 3.0 or Google will just take larger margins.
You care from a technical standpoint.
But, if you insist on going by the cost metric, you've already lost because you can buy nVidia GPUs and that's a lot cheaper than any of the TPU instances :)
But, if you insist on going by the cost metric, you've already lost because you can buy nVidia GPUs and that's a lot cheaper than any of the TPU instances :)
Cost is to do some task.
No. You don't pay google to run a task. You pay google to run a TPU instance for some amount of time. The time you spend leaving it on, setting it up, tearing it down, etc, are all still time you pay for. When they have TPU lambda jobs, that's different, but they don't. From their page:
Virtual machine pricing In order to connect to a TPU, you must provision a virtual machine (VM), which is billed separately. For details on pricing for VM instances, see Compute Engine pricing.
Virtual machine pricing In order to connect to a TPU, you must provision a virtual machine (VM), which is billed separately. For details on pricing for VM instances, see Compute Engine pricing.
You are a bit much. You have some task to complete. You can chose to use AWS with Nvidia chips or can use Google with their TPUs.
How much it cost you to complete that task is what matters. How it is done is here or there as long as get the precision.
We can see right now Google with their TPUs is about 1/2 the cost of using Nvidia with AWS.
How much it cost you to complete that task is what matters. How it is done is here or there as long as get the precision.
We can see right now Google with their TPUs is about 1/2 the cost of using Nvidia with AWS.
No, you can't. Under the best case circumstance where you have a V100 and TPU both using tensorflow for something that's optimized for both, the TPU is about 37% better:
https://blog.riseml.com/comparing-google-tpuv2-against-nvidi...
The 50% is a number you made up, and isn't based in any real benchmarks. For all other tasks that are not tensorflow tasks, the V100 is the only one that works.
https://blog.riseml.com/comparing-google-tpuv2-against-nvidi...
The 50% is a number you made up, and isn't based in any real benchmarks. For all other tasks that are not tensorflow tasks, the V100 is the only one that works.
We can see the cost and they are signficantly less for TPU 2.0. But now Google has the 3.0.
But if we look at WaveNet we can see Google must be taking much larger margins with the TPU 2.0 versus Amazon using Nvidia.
Rolling out using a NN at 16k cycles a second and offering at a competitive price to the old way means Google TPUs have to be way more efficient than using anything from Nvidia.
It is hard to believe Google pulled it off. But if we look at WaveNet it suggests that TTS is a solved problem.
It will be how it is done for a very long time and just the NN will improve.
Nvidia honestly needs to get on their game. Google is running a 1000 mph. Iterating to the TPU 3.0 in just a year was a big surprise.
I suspect Capsule networks and dynamic routing which were invented by Google drove the TPU 3.0 but do not know.
Hope Google will share a paper now on the TPU 2.0 and their secrets.
Lucky for Nvidia they will share and Nvidia can copy. But just keeps Nvidia behind.
But if we look at WaveNet we can see Google must be taking much larger margins with the TPU 2.0 versus Amazon using Nvidia.
Rolling out using a NN at 16k cycles a second and offering at a competitive price to the old way means Google TPUs have to be way more efficient than using anything from Nvidia.
It is hard to believe Google pulled it off. But if we look at WaveNet it suggests that TTS is a solved problem.
It will be how it is done for a very long time and just the NN will improve.
Nvidia honestly needs to get on their game. Google is running a 1000 mph. Iterating to the TPU 3.0 in just a year was a big surprise.
I suspect Capsule networks and dynamic routing which were invented by Google drove the TPU 3.0 but do not know.
Hope Google will share a paper now on the TPU 2.0 and their secrets.
Lucky for Nvidia they will share and Nvidia can copy. But just keeps Nvidia behind.
Oh man... You think Google is performing some kind of secret that Nvidia has no idea how to do? I really don't understand your fanboyism. The only reason the TPU can perform more operations per watt is because it is a severely limited processor. There is nothing special about that. They chose to dedicate more silicon to a smaller number of features, while Nvidia made the processor more general. If anything, Google uses the generic tsmc Fab, well Nvidia has their own subprocess at tsmc. If Nvidia really, really wanted to make a chip that was dedicated just for deep learning and nothing else, they could. But that's only useful for Google.
The big difference is Google works top down. So Google comes up with WaveNet as an example. That then causes a need to optimize the entire stack to be possible to offer at scale.
So WaveNet part of the Google Assistant and things like Duplex. But Nvidia silicon is just not going to make it possible. So Google does the TPUs as it just would not be possible to do at a reasonable price with Nvidia.
They are doing 16k cycles through a NN in real-time and competing against a far less compute intensive technique.
Now do not get me wrong I am long Nvidia and been for a while. Bit disapointed on the hit after an incredible earnings report.
I think they will do well from a investor marketing standpoint as really the only alternative to Google hardware.
But they have a fundamental disadvantage. They just do not have the applications like Google. They do NOT have the data to iterate like Google. It is why they appears to be 2 generations behind Google.
What is the goal of Nvidia? It is hardware for hardware sake? IMO, it should be driven top down and just do not see that happening at Nvidia. Is their goal the singularity?
Versus Google clearly wants to create the singularity and the silicon is in support. That is a very different calculus compared to Nvidia.
We can see it so strongly this week. The Google Keynote was all about AI applications. Then the TPUs are too support. Take a look at the duplex video for example. That is the focus and the silicon is what makes it possible.
But the more success for Google and presentations like Duplex this week and the buzz across the Internet helps Nvidia if the goal is investing. But from an actual technical solution it just points out the problem for Nvidia that much stronger.
So WaveNet part of the Google Assistant and things like Duplex. But Nvidia silicon is just not going to make it possible. So Google does the TPUs as it just would not be possible to do at a reasonable price with Nvidia.
They are doing 16k cycles through a NN in real-time and competing against a far less compute intensive technique.
Now do not get me wrong I am long Nvidia and been for a while. Bit disapointed on the hit after an incredible earnings report.
I think they will do well from a investor marketing standpoint as really the only alternative to Google hardware.
But they have a fundamental disadvantage. They just do not have the applications like Google. They do NOT have the data to iterate like Google. It is why they appears to be 2 generations behind Google.
What is the goal of Nvidia? It is hardware for hardware sake? IMO, it should be driven top down and just do not see that happening at Nvidia. Is their goal the singularity?
Versus Google clearly wants to create the singularity and the silicon is in support. That is a very different calculus compared to Nvidia.
We can see it so strongly this week. The Google Keynote was all about AI applications. Then the TPUs are too support. Take a look at the duplex video for example. That is the focus and the silicon is what makes it possible.
But the more success for Google and presentations like Duplex this week and the buzz across the Internet helps Nvidia if the goal is investing. But from an actual technical solution it just points out the problem for Nvidia that much stronger.
So how does cost compare to run a task 24/7 for a year if I buy Nvidia GPUs vs rent TPUs?
Would expect Nvidia GPUs run 24/7 would be far cheaper. But that is an unusual use case.
Even more so with ML training. It is much more bumpy.
But you have a lot of other issues. Buying is going to hurt cash flow versus renting is a big one for small companies or startups not well funded.
Then also responsible for updating and stuck with old silicon. So for example we just got the TPU 3.0. So the cost will decrease.
But your cost of running what you buy is static. You are not be rewarded by improvements and they are happening quickly with the TPUs for example.
Just one year and we have the 3.0.
Even more so with ML training. It is much more bumpy.
But you have a lot of other issues. Buying is going to hurt cash flow versus renting is a big one for small companies or startups not well funded.
Then also responsible for updating and stuck with old silicon. So for example we just got the TPU 3.0. So the cost will decrease.
But your cost of running what you buy is static. You are not be rewarded by improvements and they are happening quickly with the TPUs for example.
Just one year and we have the 3.0.
Let's see, last year I bought a 4x 1080Ti workstation for ~$6k. I agree that I haven't run it 24/7 all year long, the typical usage pattern has been training a model for 2-8 days, then it's 0-3 days idle time. The longest it ever sat idle was a week. Let's say 50% utilization rate for the last 12 months. That would be 180x24 = 4,300 hours. In a small startup, typically such workstations would be shared by multiple people, so the utilization would be higher. When I interned in a small startup people always waited to launch their experiments, so the utilization was actually close to 24/7.
So, how much would 4k hours of 4x 1080Ti compute time cost me to rent? Or, to put another way, how many hours do I need to use to justify buy vs rent? Or, how many hours of 4x 1080Ti compute time can I rent for $6k?
So, how much would 4k hours of 4x 1080Ti compute time cost me to rent? Or, to put another way, how many hours do I need to use to justify buy vs rent? Or, how many hours of 4x 1080Ti compute time can I rent for $6k?
The thing is ML for training is much more bumpy. You need further capacity at certain times.
But the other issue is you are stuck with the hardware. Google just put out the TPU 3.0 and you can use them without losing all your investment if you bought your own hardware.
The other is buying the hardware is much harder from a cash flow standpoint.
But the other issue is you are stuck with the hardware. Google just put out the TPU 3.0 and you can use them without losing all your investment if you bought your own hardware.
The other is buying the hardware is much harder from a cash flow standpoint.
So you completely ignored my questions, and simply repeated your previous comment word for word?
You provided some generic considerations, which don't apply to my situation, and I'm a fairly typical ML researcher. In fact, I know several startups doing DL research and they all have been buying hardware. Can you give me some examples where doing DL training in the cloud makes financial sense?
Also, you seem to imply that TPU is some kind of a miracle chip, even though we know very little about TPU2 and we know nothing about TPU3. It's actually pretty embarrassing that a general purpose V100 GPU is competitive with TPU2, which is an ASIC made for DL. If Nvidia ever decides to make a pure DL chip it will destroy anything Google can design. I mean, you can't be serious comparing Google to Nvidia when it comes to designing chips, right? That would be like believing that Nvidia could make a competitive search engine.
You provided some generic considerations, which don't apply to my situation, and I'm a fairly typical ML researcher. In fact, I know several startups doing DL research and they all have been buying hardware. Can you give me some examples where doing DL training in the cloud makes financial sense?
Also, you seem to imply that TPU is some kind of a miracle chip, even though we know very little about TPU2 and we know nothing about TPU3. It's actually pretty embarrassing that a general purpose V100 GPU is competitive with TPU2, which is an ASIC made for DL. If Nvidia ever decides to make a pure DL chip it will destroy anything Google can design. I mean, you can't be serious comparing Google to Nvidia when it comes to designing chips, right? That would be like believing that Nvidia could make a competitive search engine.
I do not know your personal use case. But if buying something makes sense then I would do it.
But the future is the cloud. I have fought people on this for well over a decade and thought everyone by now got it. Apparently not.
On TPU being a miracle chip not clear to me what a "miracle" chip would even be? Plus things keep improving and how could you have a static "miracle" chip?
What we do know is Google is doing WaveNet in production in real-time. We can hear the results. They are offering at a competitive price to traditional TTS.
We can also see the TPUs are about 1/2 the cost of using Nvidia.
Otherwise that is it. But I do not know your faith but personally I would not consider that a "miracle". But maybe just me.
Maybe Lewandowski would as I have heard he prays to a AI god or something like that.
The big difference is Google is working top down and Nvidia has to work bottom up.
So Google wants to roll out the best TTS there is to the world and at scale.
So they are going to from the top down to be able to achieve.
Clearly Google has a goal of creating the singularity and the Silicon is just a piece of the equation.
Versus I am not really sure what Nvidia goal is. Silicon is not for silicon sake. But rather a means to an end.
Plus Google having in production has the data to optimize in future versions where Nvidia just is not going to be able to.
I am long Nvidia and owned it for a bit. Was disappointed on how the stock traded after ER. Do think it will do fine as Google does all these incredible AI things. It is an alternative that will be good for them.
But for the very long term I would not hold. Google is a hold forever. Nvidia would keep on it.
BTW, would say Nvidia is now two generations behind.
But the future is the cloud. I have fought people on this for well over a decade and thought everyone by now got it. Apparently not.
On TPU being a miracle chip not clear to me what a "miracle" chip would even be? Plus things keep improving and how could you have a static "miracle" chip?
What we do know is Google is doing WaveNet in production in real-time. We can hear the results. They are offering at a competitive price to traditional TTS.
We can also see the TPUs are about 1/2 the cost of using Nvidia.
Otherwise that is it. But I do not know your faith but personally I would not consider that a "miracle". But maybe just me.
Maybe Lewandowski would as I have heard he prays to a AI god or something like that.
The big difference is Google is working top down and Nvidia has to work bottom up.
So Google wants to roll out the best TTS there is to the world and at scale.
So they are going to from the top down to be able to achieve.
Clearly Google has a goal of creating the singularity and the Silicon is just a piece of the equation.
Versus I am not really sure what Nvidia goal is. Silicon is not for silicon sake. But rather a means to an end.
Plus Google having in production has the data to optimize in future versions where Nvidia just is not going to be able to.
I am long Nvidia and owned it for a bit. Was disappointed on how the stock traded after ER. Do think it will do fine as Google does all these incredible AI things. It is an alternative that will be good for them.
But for the very long term I would not hold. Google is a hold forever. Nvidia would keep on it.
BTW, would say Nvidia is now two generations behind.
Clearly Google has a goal of creating the singularity
Oh wow. Didn't realize you're one of those. I used to be a Kurzweil fan too, when I was 18. That phase usually passes by senior year in college. Well, I guess some people need longer to see reality. I recommend cutting back on pop tech news consumption.
Take it easy, and have a nice day!
Oh wow. Didn't realize you're one of those. I used to be a Kurzweil fan too, when I was 18. That phase usually passes by senior year in college. Well, I guess some people need longer to see reality. I recommend cutting back on pop tech news consumption.
Take it easy, and have a nice day!
I haven't seen much info yet on TPU 3, but TPU 2 was:
- 64 TPUs per pod (11.5 petaflops)
- 4 chips per TPU (180 teraflops)
- 64 TPUs per pod (11.5 petaflops)
- 4 chips per TPU (180 teraflops)
It's 2x faster per area:
https://twitter.com/soumithchintala/status/99391656047246131...
https://twitter.com/soumithchintala/status/99391656047246131...
This is about scaling. Since you usually can't achieve 8x speed with 8 chips (there will be issues with interconnect, algorithm, etc.), 8x speed with 8 chips is noteworthy. Although I wouldn't describe that scaling work as "8 times more powerful" myself.
~8x speed with 8 chips of the same generation would obviously be very impressive. But 8x speed with 8 chips compared to a last year's single chip is definitely not the same thing.
Pretty sure you can get more than (or around) 8x speed up when comparing 8 V100 vs a single P100 for example.
Pretty sure you can get more than (or around) 8x speed up when comparing 8 V100 vs a single P100 for example.
More chips on a board requires more cooling or less power consumption.
What will be interesting to see is if they are going the hardware specialization route, like Nvidia with their support for efficient 4x4 fused FP16 matmuls with FP32 matrix output (they call them 'tensor cores' - hah).
I suspect with the liquid cooling, they are just dialing up the matmul speed, which is probably the right way to go, IMO.
We are doing data-center experiments - using oil cooling. It's suprising to see circuitboards dumped in ordinary oil and working away, no short circuits.
We are doing data-center experiments - using oil cooling. It's suprising to see circuitboards dumped in ordinary oil and working away, no short circuits.
You mean immersion cooling? As long as the fluid isn't conductive you're good to go.
If you're interested, Alibaba has been doing some work along the lines of production immersion cooling.
If you're interested, Alibaba has been doing some work along the lines of production immersion cooling.
Yes, it's not my research - a colleagues. He has some big vats of oil. It's a sureal experience to see servers just being oil-boarded (can i say that? :) ) and coming out ok!
I'd love to discuss your research. We've been working low-profile during the last years doing a lot of R&D and experimenting with different fluids and components.
Mineral oil is ok for experimentation, but for long term material compatibility and fire risk I wouldn't recommend it.
FWIW I co-founded https://submer.com where we've developed an all-embedded computing immersion cooling solution that is virtually compatible with any kind of hardware (even fiber optics) and it's orders of magnitude more efficient than traditional data center cooling technologies.
Mineral oil is ok for experimentation, but for long term material compatibility and fire risk I wouldn't recommend it.
FWIW I co-founded https://submer.com where we've developed an all-embedded computing immersion cooling solution that is virtually compatible with any kind of hardware (even fiber optics) and it's orders of magnitude more efficient than traditional data center cooling technologies.
Cooling via oil insulation isn't something new - it is used for years inside high voltage transformers to keep temperature stable.
What kind of oil are you using?
What kind of oil are you using?
mineral oil based cooling is well known in enthusiast PC builds but it leaves weird gunk on boards that is harmless but a big PITA to clean up.
Also - some oils while won't hurt your circuits, they might start dissolving the various plastic connectors on cables and board.
Also - some oils while won't hurt your circuits, they might start dissolving the various plastic connectors on cables and board.
Does the Google Cloud API for TensorFlow expose any hardware details? I am just curious why Google announces these hardware details at all.
Look at this thread. Because people want to know as just what "techies" do.
But what I wanted to know is performance in terms of joules compared to TPU 2s.
My hope is we get a paper on the TPU 2 now the 3 is released. We got the TPU 1 paper as they were releasing the TPU 2 I suspect.
But what I wanted to know is performance in terms of joules compared to TPU 2s.
My hope is we get a paper on the TPU 2 now the 3 is released. We got the TPU 1 paper as they were releasing the TPU 2 I suspect.
Is this going to be available off-the-shelf like NVidia GPUs? That's the only way to get wider and faster developer buy-in.
No, they're going to be used by Google and nobody else, so they can establish dominance in the cloud AI/AI SaaS space using their immense resources. This will be used by internal Google projects to rapidly develop their large scale models and, if you're lucky, available on Google Cloud one day (TPUv2 already is, at least).
Google doesn't need "developer buy in" for these to make sense -- they need better hardware for training and deploying deep learning models for their products, which is the overwhelming motivation. And, if they offer these to you, it's only because you're willing to pay for faster TensorFlow iteration.
Google doesn't need "developer buy in" for these to make sense -- they need better hardware for training and deploying deep learning models for their products, which is the overwhelming motivation. And, if they offer these to you, it's only because you're willing to pay for faster TensorFlow iteration.
Models are the new bitcoins and no one is willing to sell the hardware directly. Weirdly enough exact same thing happened in mining (specialized hardware passed nvidia gpus in performance at some point).
They did denied selling older generation, so I would expect TPU 3 is not going to be available as well. The only way to use those is to use Google Cloud.
Would doubt it and not really make much sense in 2018.
No sense at all. Nvidia's stock increased 14x since 2014 by running a closed system in a public cloud. Oh wait...
Not sure what it has to do with Nvidia stock. The future is cloud providers using their own silicon.
Google has set the bar where others will have to match. Also looks like Google is now 2 generations ahead of Nvidia.
Google has set the bar where others will have to match. Also looks like Google is now 2 generations ahead of Nvidia.
Nvidia is on their 5th generation of datacenter ASIC. Google is on their third. Nvidia has a faster interconnect than anyone else (nvswitch), which allows close to the same access bandwidth to HBM on a remote GPU than a local GPU. If Google has that, they haven't announced it, and all I can see is they operate over PCIe v3 since they are slaves to the CPU manufacturers. If the future was cloud providers using their own silicon, Google would have made a CPU to compete with Intel. Instead, they use Intel exclusively and are evaluating POWER. They made a single ASIC that is very simple relatively speaking, that can perform a single task and cannot be accessed generically. They are not 2 generations ahead. They chose a path that's much easier for many reasons, but mostly because it allows their own internal training to be accelerated.
All comes down to the cost. We can see that the TPUs were about 1/2 as much as using Nvidia before the TPU 3.0. This is based on doing the same task with the TPUs versus Nvidia with AWS. But it might be a bigger difference and Google taking margins. Could have been the other way but think WaveNet proves not that direction.
Would hope we get an even bigger price difference with this generation.
BTW, you are mixing up graphic generations and ML generations. Google never did or will graphics.
For ML Google is 2 generations ahead. Hard to imagine Nvidia ever catching up as so much of the AI breakthroughs are coming from Google.
A perfect example is WaveNet. Google is using a NN for audio in real time at 16k cycles a second. That is competing with using the old way of doing this. The Google approach gets a better result but people are only willing to pay too much more for a better result.
So Google had to price WaveNet competitively to the old approach which they have. I suspect that would have been impossible using Nvidia. The cost in running would have been just too high. The computation required for WaveNet is huge and the old way it is minimal.
Also we are going to need to really separate training with inference. They have different requirements and the costs in running are different.
Has Nvidia done an inference only solution? Or do they still always mix them?
"Google would have made a CPU to compete with Intel. "
I do a lot of surfing and have to say one of the more silly things to read. There is little gains today with CPUs. Why on earth would you invest in doing your own?
Google has ported all their stuff to Power so they have a backup and not reliant on Intel. But CPUs are NOT the future and would be crazy to invest into them at this point.
Processing is moving to TPU type processors and we will see more and more traditional things. A fantastic paper on this from Jeff Dean at Google I suggest you read.
https://arxiv.org/abs/1712.01208
One thing I love about Google is they do NOT reinvent the wheel. If there is something that can work like the Linux kernel they use. Versus having an ego of having to do yourself.
They did all their own network silicon by hiring the Lanai team several years ago because there was nothing that could work for their needs. They then created a determinate network stack so they could do Spanner. There was nothing off the shelve. Heck they laid their own fiber under the ocean to make possible.
Spanner is the first and only horizontally scalable RDBMS solution to exist. Google beat the speed of light by using their custom silicon, custom stack that is determinate, their own fiber. Then using atomic clocks with GPS to remove the latency from speed of light out of the equation. Highly recommend the papers including the Spanner paper.
https://ai.google/research/pubs/pub39966
The network uses custom silicon inside and then they put the intelligence on out edge. Then they control exactly the traffic so they get a determinate result. This also allows them to use far cheaper hardware as do not need to over provision as they never drop packets on the ground or need much memory for buffers.
Google would have used Nvidia in a second if they could do what they need.
But what is being done in AI (ML) today requires unique silicon. Just not possible without as the power footprint would be prohibitive.
I do expect Google more and more to leverage on client ML where needed with their PVC. So it is NOT just the TPUs but for somethings you are going to need on client.
Take the new voices as an example. They are doing 6 voices as each has a different model. The cost in switching models is prohibitive in the cloud.
So you will see them eventually move to some things being done on the client. But they are so far ahead they have tons of time.
I would not be surprised if we do not seen anyone else do Wavenet for a couple of years. What Nvidia should be working on is finding a way to do WaveNet at a reasonable cost for Amazon and others.
But the long term is the big cloud providers will do their own silicon. Google set the path with the TPUs and now the others will have to do the same to keep up. But just makes sense as the cloud providers have the data to iterate.
MS is using a FPGA which was foolish, IMO. Never going to get there with that route.
BTW, you need to realize how this works for inference. You keep the model in memory.
Google invented Capsule networks that use dynamic routing and are going to cause different patterns in memory access and would be curious if the TPU 3.0 were optimized for such?
They have a huge advantage of inventing the algorithms and then get to make their own silicon optimized.
Something Nvidia really needs to be able to better compete, IMO.
BTW2, Google using Power did include the excellent Nvidia memory interconnect as part of the spec.
Not sure did Google license or does Nvidia give away?
The spec is called Open Power if not aware?
https://www.theregister.co.uk/2016/04/07/open_power_summit_p...
Would hope we get an even bigger price difference with this generation.
BTW, you are mixing up graphic generations and ML generations. Google never did or will graphics.
For ML Google is 2 generations ahead. Hard to imagine Nvidia ever catching up as so much of the AI breakthroughs are coming from Google.
A perfect example is WaveNet. Google is using a NN for audio in real time at 16k cycles a second. That is competing with using the old way of doing this. The Google approach gets a better result but people are only willing to pay too much more for a better result.
So Google had to price WaveNet competitively to the old approach which they have. I suspect that would have been impossible using Nvidia. The cost in running would have been just too high. The computation required for WaveNet is huge and the old way it is minimal.
Also we are going to need to really separate training with inference. They have different requirements and the costs in running are different.
Has Nvidia done an inference only solution? Or do they still always mix them?
"Google would have made a CPU to compete with Intel. "
I do a lot of surfing and have to say one of the more silly things to read. There is little gains today with CPUs. Why on earth would you invest in doing your own?
Google has ported all their stuff to Power so they have a backup and not reliant on Intel. But CPUs are NOT the future and would be crazy to invest into them at this point.
Processing is moving to TPU type processors and we will see more and more traditional things. A fantastic paper on this from Jeff Dean at Google I suggest you read.
https://arxiv.org/abs/1712.01208
One thing I love about Google is they do NOT reinvent the wheel. If there is something that can work like the Linux kernel they use. Versus having an ego of having to do yourself.
They did all their own network silicon by hiring the Lanai team several years ago because there was nothing that could work for their needs. They then created a determinate network stack so they could do Spanner. There was nothing off the shelve. Heck they laid their own fiber under the ocean to make possible.
Spanner is the first and only horizontally scalable RDBMS solution to exist. Google beat the speed of light by using their custom silicon, custom stack that is determinate, their own fiber. Then using atomic clocks with GPS to remove the latency from speed of light out of the equation. Highly recommend the papers including the Spanner paper.
https://ai.google/research/pubs/pub39966
The network uses custom silicon inside and then they put the intelligence on out edge. Then they control exactly the traffic so they get a determinate result. This also allows them to use far cheaper hardware as do not need to over provision as they never drop packets on the ground or need much memory for buffers.
Google would have used Nvidia in a second if they could do what they need.
But what is being done in AI (ML) today requires unique silicon. Just not possible without as the power footprint would be prohibitive.
I do expect Google more and more to leverage on client ML where needed with their PVC. So it is NOT just the TPUs but for somethings you are going to need on client.
Take the new voices as an example. They are doing 6 voices as each has a different model. The cost in switching models is prohibitive in the cloud.
So you will see them eventually move to some things being done on the client. But they are so far ahead they have tons of time.
I would not be surprised if we do not seen anyone else do Wavenet for a couple of years. What Nvidia should be working on is finding a way to do WaveNet at a reasonable cost for Amazon and others.
But the long term is the big cloud providers will do their own silicon. Google set the path with the TPUs and now the others will have to do the same to keep up. But just makes sense as the cloud providers have the data to iterate.
MS is using a FPGA which was foolish, IMO. Never going to get there with that route.
BTW, you need to realize how this works for inference. You keep the model in memory.
Google invented Capsule networks that use dynamic routing and are going to cause different patterns in memory access and would be curious if the TPU 3.0 were optimized for such?
They have a huge advantage of inventing the algorithms and then get to make their own silicon optimized.
Something Nvidia really needs to be able to better compete, IMO.
BTW2, Google using Power did include the excellent Nvidia memory interconnect as part of the spec.
Not sure did Google license or does Nvidia give away?
The spec is called Open Power if not aware?
https://www.theregister.co.uk/2016/04/07/open_power_summit_p...
100 petaflops per what? All of google’s deployment? Per rack? 100 Pflop / chip can’t be right.
Yeah I love this, 8x PFlop improvement per pod, but the pod size has increased 4x. That 8x speed up is now mystically halved.
*mystically quartered.
Per pod (256 TPUs)
Can you allocate a whole pod to do your computations? How much does that cost?
They increased the pod size by 4X, also each "TPU" has 4 tpu chips on it IIRC.
This very typical google. V100 comes out, they don't deploy it in their cloud immediately. Then they launch their tpu cloud. Spend 2 months touting cost/speed in benchmarks. Then a week before io they make v100 available in gce, nearly six months after aws. Then at io they announce tpu3. This is supposed to be nvidia's most captive customer, and now looks like going to be their biggest problem. Would love to see google spend with them and how its changed since they ramped tpu2.
Disclosure: I work on Google Cloud (and even on the GCE parts).
No conspiracy here, we were simply late to market on the V100 Beta. As many have noted, we also hold a really high reliability bar before going to Beta (“Google’s Beta is like AWS’s GA” though I wouldn’t claim that in all cases) and we’ve be in Alpha for months. If you’d like to be included in Alpha offerings drop us a line.
As I’ve said in previous threads: Compute Engine intends to be the best place for computing. That includes the kind of workloads TPUs are tuned for, but it also includes all the stuff that GPUs excel at. That’s not going away, and GCE isn’t favoring one over the other. We sell infrastructure, and we don’t try to (overtly) pick favorites, we let our customers do that.
No conspiracy here, we were simply late to market on the V100 Beta. As many have noted, we also hold a really high reliability bar before going to Beta (“Google’s Beta is like AWS’s GA” though I wouldn’t claim that in all cases) and we’ve be in Alpha for months. If you’d like to be included in Alpha offerings drop us a line.
As I’ve said in previous threads: Compute Engine intends to be the best place for computing. That includes the kind of workloads TPUs are tuned for, but it also includes all the stuff that GPUs excel at. That’s not going away, and GCE isn’t favoring one over the other. We sell infrastructure, and we don’t try to (overtly) pick favorites, we let our customers do that.
Apologies, stand corrected. Conspiracy theories are always more fun:) Guess was just coincidence that after grabbing all this press on GCE v100 finally being available, tpu3 is announced few days later.
[deleted]
> “Google’s Beta is like AWS’s GA” though I wouldn’t claim that in all cases
Wow, that's so ... I don't even have a word for it.
Wow, that's so ... I don't even have a word for it.
Lower GPU price for all, I am for it.
For all the google hate, nvidia is too blame as well. Their ceo running around saying 'we saving you money every time you spend 10k on a gpu' is getting old. The should have been far more aggressive price wise with this biz considering the resources of their target customers. Almost every 'choose not to use a gpu' use case being touted by msft/goog etc is about cost. Meanwhile nvda busy tweeting stuff like msft seeing ai app for blind powerd by nvda gpu. Soon micron will be tweeting they powering all ai as well, and then arista...all the way down to the power companies. Watching goog today its not looking good for the competition, by the time all these ml/dl accelerators are commoditized google will have one hell of a moat.
You do not want to be dependent on Nvidia for your own stuff is why Google has to do the TPUs.
It is like how they ported to Power for their stuff if needed.
Look at the shortage of late with Nvidia. Google does not have to deal with it.
But also I suspect they could not do WaveNet without their own silicon as the cost would be prohibitive. Using a NN for speech and competing against traditional approaches and have a competitive price is going to require really efficient silicon.
It is like how they ported to Power for their stuff if needed.
Look at the shortage of late with Nvidia. Google does not have to deal with it.
But also I suspect they could not do WaveNet without their own silicon as the cost would be prohibitive. Using a NN for speech and competing against traditional approaches and have a competitive price is going to require really efficient silicon.
What shortage? You can't make statements like that with no basis in reality. The V100 is not suffering from any shortage.
"Looking to nab Nvidia's GeForce chips? You need cash and patience
GPU shortage equals four-month wait time for buyers"
https://www.theregister.co.uk/2018/03/30/nvidia_geforce_chip...
But you are missing my point. Google is not dependent on Nvidia.
Really does not look like Nvidia could do what they need anyway. Doing their new text to speech on Nvidia I would not expect to have a price point to make viable.
GPU shortage equals four-month wait time for buyers"
https://www.theregister.co.uk/2018/03/30/nvidia_geforce_chip...
But you are missing my point. Google is not dependent on Nvidia.
Really does not look like Nvidia could do what they need anyway. Doing their new text to speech on Nvidia I would not expect to have a price point to make viable.
So let me get this straight. You are saying Nvidia is having a massive shortage of parts, while ignoring that both AWS and GCP offer GPU instances that you can get any time. Not only that, the article you reference is about Geforce GPUs, which is not a data center GPU. You are comparing apples and oranges.
Please show me an article where there's a shortage of Tesla cards.
Please show me an article where there's a shortage of Tesla cards.
It is about being dependent on someone else for something that is critical to you.
But really now it is just Nvidia does not have the ability to meet Google's needs.
I am not aware of anything from Nvidia that could do 16k cycles through a NN in real-time at a cost that you could roll out at scale.
Google priced WaveNet competitively to the old way of doing. You can only do that with a very low cost for computation and the power required which Nvidia could not match with the TPU 1.0 let alone now they have the 3.0.
But really now it is just Nvidia does not have the ability to meet Google's needs.
I am not aware of anything from Nvidia that could do 16k cycles through a NN in real-time at a cost that you could roll out at scale.
Google priced WaveNet competitively to the old way of doing. You can only do that with a very low cost for computation and the power required which Nvidia could not match with the TPU 1.0 let alone now they have the 3.0.
Spend with them? Who?
Their spend on nvidia gpu's...how it's shifted as they have progressed tpu wise. I think you go back 18 months google was their biggest here.
But does it not make sense for Google to remove their dependency on Nvidia?
Plus could they do the new text to speech at a competitive price using Nvidia Silicon?
Google approach gets you a better result but the computational requirements would be huge compared to the traditional approach. You have to get the compute cost down as far as you can.
Plus could they do the new text to speech at a competitive price using Nvidia Silicon?
Google approach gets you a better result but the computational requirements would be huge compared to the traditional approach. You have to get the compute cost down as far as you can.
The announcement bit on video https://youtu.be/ogfYd705cRs?t=1h36m
[deleted]
Welcome to ASIC land....training going the route of crypto mining
It's just an arithmetic logic unit for tensors. It's not at all like a crypto miner that implements a single algorithm.
You kind of summed it up right there...'it's just an arithmetic logic unit for tensors'...whether its a matrix multiply engine or implmenting sha256 still a custom rapidly iterated chip for narrow very specific use case. Google accomplishment here clearly the software, but doubt they only ones going to crack asic systolic arrays. At some point china inc figures this out in mass...maybe bitmain themselves.
Bitmain already announced TPUs (cue "they'll only sell them after they're finished training Skynet").
FYI, here is Bitmain's AI product homepage: https://www.sophon.ai/
They shipped Deep Learning Accelerating Card SC1 at $589. You can't buy it right now because it is sold out.
They shipped Deep Learning Accelerating Card SC1 at $589. You can't buy it right now because it is sold out.
Well... I'll concede that 2-instructions that aren't reading or writing memory is a pretty narrow set of operations. But it's also much more generally useful than computing one specific hash function.
Completely agree with you there. Guess the analogy I was making was more about the evolutionary speed we saw there. Google seems to be moving awful fast here.
Faster matrix multiplication is something which has application in practically every engineering field there is.
Exactly. Drives me crazy when people compare to a single alogrithm chip. It is not.
But the TPU actually is: it's a systolic array matrix multiply ASIC.
The TPUs support all kinds of NN. Not specific to one type.
Indeed, all neural nets that use matrix multiplication. Out just so happens to be all the main types right now.
Google is the leader on AI algorithms. Where GANs and Capsules and so many others came from for example. So things change and Google can implement in silicon far earlier than anyone else as needed.
But right now they have what they need.
BTW, I would be curious on the TPU 3 design considerations to better support dynamic routing with Capsule networks. I suspect creates different memory access patterns and where the big power savings are at today.
But right now they have what they need.
BTW, I would be curious on the TPU 3 design considerations to better support dynamic routing with Capsule networks. I suspect creates different memory access patterns and where the big power savings are at today.
You are constantly comparing this single-purpose chip to a general-purpose GPU. How is that any different?
Just to play devil's advocate for a moment: I'm excited that there will finally be some viable competition for GPUs, but am disappointed that this isn't a general-purpose multiprocessor.
We're long overdue for a general-purpose CPU with say 1024 cores, that avoids a central main memory, where each core can be independently programmed just like any other CPU. Google's may count as a somewhat general-purpose DSP, which is definitely a step forward. But no matter how mature or mainstream a framework like TensorFlow gets, it can never replace full programmability.
Without seeing the internals, I'm going to have to give this a nay vote for now. There are many other rather exciting problems that need to be opened up to a new generation of tinkerers. Off that top of my head, it's things like: content-addressable memory to provide high data locality (evolving various interconnects instead of hardwiring them), exploring other types of general vector processing like the kind MATLAB/Octave uses, and exploring other hill-climbing algorithms than backpropagation/neural nets.
I picture something more like network topology-agnostic Docker containers programmed in Elixer/Erlang/Go that can act as semi-autonomous agents and switch into various modes in order to solve the problem at hand. I just find that a much simpler metaphor to work with than OpenCL/CUDA/TensorFlow. Yes it would take more silicon and would probably violate YAGNI, but only full programmability gives us the freedom to explore the problem space at the level that's going to be required to implement artificial general intelligence.
We're long overdue for a general-purpose CPU with say 1024 cores, that avoids a central main memory, where each core can be independently programmed just like any other CPU. Google's may count as a somewhat general-purpose DSP, which is definitely a step forward. But no matter how mature or mainstream a framework like TensorFlow gets, it can never replace full programmability.
Without seeing the internals, I'm going to have to give this a nay vote for now. There are many other rather exciting problems that need to be opened up to a new generation of tinkerers. Off that top of my head, it's things like: content-addressable memory to provide high data locality (evolving various interconnects instead of hardwiring them), exploring other types of general vector processing like the kind MATLAB/Octave uses, and exploring other hill-climbing algorithms than backpropagation/neural nets.
I picture something more like network topology-agnostic Docker containers programmed in Elixer/Erlang/Go that can act as semi-autonomous agents and switch into various modes in order to solve the problem at hand. I just find that a much simpler metaphor to work with than OpenCL/CUDA/TensorFlow. Yes it would take more silicon and would probably violate YAGNI, but only full programmability gives us the freedom to explore the problem space at the level that's going to be required to implement artificial general intelligence.
NVIDIA is now exerting pricing power to the point where they’ve decided to train their sales people to disregard the metric that is most important to customers: cost of training. Talk with one of their enterprise sales people and you’ll find they’ll say things like “FLOPS / $ doesn’t matter” to justify a 10x increase in price for their TESLA line. As history has shown, a monopolist sows the seeds of their own destruction and, by disregarding the metrics that matter, they alienate their customers.
Here are open source projects that you can contribute to to break the monopoly:
ROCm: https://rocm.github.io/
MIOpen: https://github.com/ROCmSoftwarePlatform/MIOpen
TensorFlow: https://github.com/tensorflow/tensorflow