I get that these bullets points are answering What instead of Why but for those that are more readily discernible, like "In a year-and-a-half, the time required to train a large image classification system on cloud infrastructure has fallen from about three hours in October 2017 to about 88 seconds", what's causing this? Are models getting smaller without a loss in accuracy? Is training distributed over a greater amount of cheaper machines? Personally, I'd be more excited about the former rather than the latter. We can't all afford MegatronLM-type experiments - https://nv-adlr.github.io/MegatronLM.
As an engineer, this makes me wish there were a better path to FANG employment than hacking their whiteboarding interviews by leetcoding for weeks. They probably do this because it's more objective and simpler than something like the "long conversation with a professor" test that pg suggests.
Who's excited about this? What's your use case that just became viable because of it? Definitely don't mean these questions in a condescending way, just want to get a read on the pulse from the folks here that will use it :)
Cost can be a killer here though. If you're flipping from blue to green and vice versa you either have to have capacity in stand-by(expensive) or spin up new capacity before flipping(time-consuming).
Understanding a sentence is fundamentally different from recognizing an object. But people are trying to use deep learning to do both.
I agree with most of the article but I think this^^ skips over the different types of networks used to solve perception and language problems. A CNN is very different from say, word2vec, which isn't a very deep network at all.
Maybe this is a good time for university researchers to develop AI algorithms that are not so data and compute hungry. Here's a promising bit of that--https://www.csail.mit.edu/news/smarter-training-neural-netwo.... This is easier said than done but necessity is the mother of all invention they say.
Great article, I liked that it illuminates the question--whats a trade secret vs knowledge from on-the-job-experience? And where's the line across which a company can say you've used a trade secret? It's scary to think that one of the outcomes of this case could be a precedent that allows companies to go after what's in your head.