4 of 5 of the biggest tech company founders were in their early 20s. So the biggest ideas clearly slanted towards young visionary founders, but the statistics do skew back towards older founders for all but the largest companies.
> When code is written only once and most of the cost comes afterward, that seems like an impossible choice to defend.
> I can't imagine how anyone working on large code bases with other people would want to do this. Yes, implicitness is more fun and beautiful at the beginning, but it becomes a nightmare after a short time for anyone other than the original coder.
Good points, it seems like the arguments for implicitness may have been stronger in the past, when programming languages were less developed. Think of RollerCoaster Tycoon being written almost entirely in assembly in '99 by a single programmer. You'd have plenty of incentive for implicit standards. When you have modern languages with well optimized abstractions, all that implicitness ends up losing out. But if you compare the total amount of work that went into, say, Rust, with the cost of a single dude just building an awesome game, you see that explicitness only wins when it gets to cheat and use way more resources. So yes, explicitness is always better in the limit, but when resources are more constrained, implicitness is so nimble it will just crush the competition.
Look, it sounds like your implying compute colocated storage in the analog properties of your system (which is exactly what a synaptic weight is btw), on top of using extremely low bit precision. So explicitly calling your system totally non-neuromorphic is a little deceiving. But even then I find this idea that you're going to be running the AlexNet communication protocol to pass around information in your system to be a little strange. If you're doing anything like passing digitized inputs through a fixed analog convolution then you're not going to beat the SRAM limit, which means that instead you have in mind keeping the data analog at all times, passing it through an increasing length of analog pipelines. Even if you get this working, I'm quite skeptical that by the time you have a complete system, you'll have reduce communication costs by even half the reduction you achieve in computation costs on a log scale. It's of course possible that I'm wrong there (and my entire argument hinges on the hypothesis that computation costs will fall faster than communication - which is true for CMOS but may be less true for optical), but this is really the only projection on which we disagree. If I'm right, then regardless of whether you can hit 50 Tops (or any value) on AlexNet, you'd be foolish not to reoptimize the architecture to reduce communication/compute ratios anyway.
In an integrated system at 50 tops/watt? How are you going to even access memory at less than 20 fJ per op? Like, you're specifically trying to hide the catch here. If we were to take you at face value, we'd have to also believe that Nvidia is working on an energy optimized system that is 50x worse for no good reason.
For reference, reading 1 bit from a very small 1.5kbit sram, which is much cheaper than the register caches in a gpu, costs more than 25 fJ per bit you read.
>Which existing neuromorphic computers achieve 10^14 ops/s at 20 W? If you compare them to GPUs, those "ops" better be FP32 or at least FP16.
The comparison is of 3 bit neuromorphic synaptic ops against FP8 pascal ops. That factor is important (as it means that the neuromorphic ops are less useful), but it turns out to be dwarfed by the answer to your second question:
> Also, you forgot to tell us what is that "extremely concrete reason why current neural net architectures will NOT work with the above optimizations".
this is rather difficult to justify in this margin. But the idea is that proposals such as those above (50 Tops) tend to be optimistic on the efficiency of the raw compute ops. But these proposals really don't have much to say about the costs of communication (e.g. reading from memory, transmitting along wires, storing in registers, using buses, etc.). It turns out that if you don't have good ways to reduce these costs directly (and there are some, such as changing out registers for SRAMs, but nothing like the 100x speedup from analog computing), you have to just change the ratio of ops / bit*mm of communication per second. There are lots of easy ways to do that (e.g. just spin your ops over and over on the same data), but the real question is how to get useful intelligence out of your compute when it is data starved. This is an open question, and (sadly), very few ppl are working on it, compared to say low-bit-precision neural nets. But I predict this sentiment will be changed over the next few years.
Edit for below: no one is suggesting 50 Top/w hardware running alex net software to my knowledge (though would love to hear what they are proposing to run at that efficiency) . Nvidia among others are squeezing efficiency for cv applications with current software, but this comes at the cost of generality (it's unlike the communication tradeoffs they're making on that chip will make sense for generic AI research), and further improvements will rely on broader software changes, esp revolving around reduced communication. There are a lot of interesting ways to reduce communication without sacrificing performance, such as using smaller matrix sizes, which would reverse the state of the art trends.
Sure - I guess it's productive for me to answer why this doesn't disagree with my comment. By the time you get the software to hook up that kind of low bit precision (READ: neuromorphic) compute performance with extreme communication-minimizing strategies (READ: neuromorphic), which will invariable require compute colocated, persistent storage (READ: neuromorphic) in any type of general AI application, you're not exactly making the argument that neuromorphic chips are a bad idea.
We literally have to start taking neuromorphic to mean some silly semantics like "exactly like the brain in every possible way" in order to disagree with it.
Edit: also, to ground this discussion, there are extremely concrete reason why current neural net architectures will NOT work with the above optimizations. That's the primary motivation for talking about "neuromorphic", or any other synonym you want to coin, as fundamentally different hardware. AI software ppl need to have a term for hardware of the future, which simply won't be capable of running AlexNet well at all, in the same way that a GPU can't run CPU code well. I think the term "neuromorphic" to describe this hardware is as productive as any.
> Yes, I totally agree. Yann LeCun, Geoff Hinton, Jurgen Schmidhuber and others did unpopular work for a long time.
...
> Until then, I'll be ... rolling my eyes at brain analogies.
Maybe you don't realize this, but these guys made more brain analogies than you can count over the same period to which you attribute their greatness. Meanwhile, they were attacked year after year by state-of-the-art land grabbers saying the same things you just did.
> isn't being presented as basic research on a risky hypothesis.
It is basic research, but it's not a risky hypothesis. Existing neuromorphic computers achieve 10^14 ops/s at 20 W. Thats 5 Tops/Watt. The best GPUs currently achieve less than 200 Gops/Watt. Where is the risk in saying that a man-made neuromorphic chip can achieve more per dollar than a GPU. There is no risk, and suggesting that this field is somehow has too much risk for advances to be celebrated is absolutely crazy.
We certainly have enough compute at this point. 10^15 flops should be more than enough to run the brain by pretty much any analysis. Part of the issue is that evolution had at least 1 million such creatures over 52,000,000 weeks to improve since monkeys. So while human intelligent design of AI will certainly be a better algorithm than evolution, we may actually be a bit shy computationally of easily training an AI system, in spite of having more than enough to realize one.
Yes! Sometimes you know that a solution will take a particular mathematical form, without knowing what the parameters will be. So you can write down a program (function) that can express any solution of that form, and use an optimization algorithm e.g. gradient descent on labeled examples, to figure out which specific instance of your possible solutions works best.
I always come to these hyperloop criticisms expecting to find some sort of fatal flaw in the physics of energy efficient supersonic travel. But to my surprise, they instead tend to be pessimists saying things like, "You'll never get past my friends: the regulators, the government bureaucrats, and especially the lawyers!. We will drive up your costs and make you look foolish".
First, no one said that designing this thing in the USA means it has to be deployed in the USA. Countries without common law legal systems get around these unnecessary costs much easier.
Second, if these are seriously the only objections, then thank god we are actually building this thing. I could see complaints if it were some $100 billion publicly funded project, but the fact that less than $1 billion in private capital has already gone so far into demonstrating the technological feasibility of such an innovative transportation mechanism is a huge win.
Just visited the Palo Alto Apple store and I'm pretty sure a Microsoft Surface Tablet has a better keyboard than the new MacBook Pro. They really smushed down the keyboard to make the whole configuration thinner, but whereas the old Pro's were halfway between the quality of a dome keyboard and a mechanical keyboard IMO, the new ones are even worse than domes.
Another interesting scenario is that Satoshi is working directly with Craig in order to lend his authority on the little/big-blockian debate to Craig. This would explain the mountain of evidence as well as Craig's apparent incompetence.
Of course this conspiracy has a low prior probability due to it's complexity, but it does explain why the inventor of bitcoin would be 1) taking of screenshots of buggy code in notepad, 2) using factorial notation to enumerate combinatorics of bitstrings, and 3) being so generally dislikable.
I think this really hits the nail on the head. "The later investors" here really include everyone except for YC, insofar as investors have heretofor been able to take the YC stamp on a company as certification that these sort of issues have been worked out. I think the end result of this is that you'll see a market correction against YC companies as VC's find they have to put in extra due diligence they didn't have to before. Altman raising attention to this issue may act as a catalyst.
OP here. I made this because I love mosh [1], but was simply unable to convince IT to let me use it on the network. Using mosh sometimes and ssh other times was a real pain for muscle memory, so I figured there must be some way of getting those mosh features without going through UDP port 60000. I was recommended to try autossh, but discovered that was lacking in polish, and wrote this instead. Hope you guys enjoy it too!
Well I think my main point was just to say that the "hypocrisy" you mention actually has a rational backing. Not that the backing is bulletproof, but it's certainly more than a hypocrisy.
You seem to most strongly disagree with the assertion that "the government is incentivized to put as many people in jail as possible". It's pretty clear from my original comment that "the government" is referring to law enforcement agencies, such as the FBI. Note that the FBI has thousands of agents whose performance is measured ultimately by the percent of cases they close. Thus, the claim that FBI agents "are incentivized to put as many people in jail as possible" is more an observation than some crackpot theory. It doesn't mean that we should change that - running a law enforcement agency any other way wouldn't make much sense. But it does provide the rational backing for someone to be more concerned about worst-case government abuse of data than worst-case corporate abuse.