Mixture-of-Expert (MoE) was introduced in the 1990s [1, 2], see also
[3, 4]. The idea was that MoE scales up model capacity and only
introduces small computation overhead. MoEs did not become viable for high-performance
applications until sparse routing was integrated with modern deep
networks, made possible by large-scale distributed computation. The
breakthrough came with the development of sparsely gated networks [5],
which showed that it is possible to maintain model accuracy while
activating only a small fraction of a large parameter network during both
training and inference.
[1] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, G. E. Hinton, Adaptive mixtures of local experts. (1991)
[2] M. I. Jordan, R. A. Jacobs, Hierarchical mixtures of experts and the EM algorithm. (1993)
[3] L. Xu, M. Jordan, G. E. Hinton, An alternative model for mixtures of experts. (1994)
[4] S. Waterhouse, D. MacKay, A. Robinson, Bayesian methods for mixtures of experts. (1995)
[5] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, J. Dean, Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. (2017)
Indeed, that is the standard approach. It is also how some of the aforementioned languages desugar opaque type synonyms during compilation. It has the slight disadvantage that we can no longer use variables like
x
in some situations, but need to use
x._polynomials_gf_2
or whatever is the structure's field name. It is nice to avoid this boilerplate, which can become annoying quickly. Let the type-checker not the human do this work ...
> You do not need another language for this.
By the Church-Turing thesis you never need another language, but empirical practise has shown that the software engineering properties we see with real-world code and real-world programmers differ significantly between languages.
There are even more algebras on the same bits, when you take signed integers into account, such as saturating arithmetic.
One interesting programming language construct that might be useful in this context are Opaque Type Synonyms, a refined form of C's typedef, which modern languages like Rust, Haskell, Go or Scala offer. This allows the programmer to use the same underlying types (e.g. int), give it different names, and define different algebras with the alias. The typing system prevents the different aliases accidentally to flow into each other. Of course that alone does not help to manage the profusion of algebras over the same bits. I think a better approach for a high-level programming language is to follow assembly and really use different names for different operations, e.g. not have + build in. Instead use explicit names like add_uint32, add_polynomials_gf_2,
add_satur_arith, etc etc. The user can then explicitly define (scoped) aliases for them, including +, as long as the typing system can disambiguate the uses. The Sail DSL for ISA specification (https://github.com/rems-project/sail) does this, and it is nice.
Yes, that's true, but the registers themselves are untyped, what modern CPUs really implement is multiple instruction semantics over the same bit-patterns. In short: same bits, five algebras! The algebras are given by different instructions (on the same bit patterns).
Here is an example, the bit pattern 1011:
• as a non-negative integer: 11. ISA operations: Arm UDIV, RISC-V DIVU, x86 DIV
• as an integer residue mod 16: the class [11] in Z/16Z. ISA operations: Arm ADD, RISC-V ADD/ADDI, x86 ADD
• as a bit string: bits 3, 1, and 0 are set. ISA operations: Arm EOR, RISC-V ANDI/ORI/XORI, x86 AND.
• as a binary polynomial: x^3 + x + 1. ISA operations: Arm PMULL, RISC-V clmul/clmulh/clmulr, x86 PCLMULQDQ
• as a binary polynomial residue modulo, say, x^4 + x + 1: the residue class of x^3 + x + 1 in GF(2)[x] / (x^4 + x + 1). ISA operations: Arm CRC32* / CRC32C*, x86 CRC32, RISC-V clmulr
And actually ... the floating point numbers also have the same bit patters, and could, in principle reside in the same registers. On modern ISAs, floats are usually implemented in a distinct register file.
You can use different functions in C on the bit patterns we call unsigned.
Are China's demographics appreciatively different from other industrialised countries? Questionable. Not to mention that it's unclear why an aging society is a problem at least for the next 100 years or so (e.g. most violent crime is perpetrated by the under-30s). It's also easy, at least for a dictatorship, to increase the birth rate (e.g. restrict access to all birth control, give massive preferences to families with children, like housing, salary, "bachelor tax" etc).
> military containment of China.
That is expensive. One frequently cited explanation for the collapse of the Soviet Union was that its poor economy could not support its oversized army, which it needed to keep the Soviet Bloc at heel (not to mention all the revolutions and secession movements it fostered elsewhere).
Also, does anyone actually believe that China is expansionist? Compare:
> In terms of design, China has world class companies.
And not just companies. This is currently the world's top open-source RISC-V development https://github.com/OpenXiangShan coming from the Chinese Academy of Science.
(Whether China will remain investing in RISC-V, given that the US government has started to pressure US RISC-V development to limit their involvement with China is another question.)
1. Stopping technology transfer worked exceedingly well for the west weakening the Soviet Union during the Cold War.
2. A distinct lack of (non-violent) alternatives for the West preventing China becoming the world's leading technological superpower (and hence also strongest military).
I doubt this will be successful in the long run, because China is not burdened by the Soviet Union's extremely inefficient way of organising its economy. Not to mention that China is the worlds biggest market.
I recommend [1] as an introduction to the semiconductor physics behind the Rowhammer problem. Rowhammer is an instance of the "weird machine" problem behind many security problems, i.e. a mismatch between two abstractions: the abstraction we pretend describes the system, vs the reality of the system. In the case of Rowhammer, that is the abstraction of memory as a digital device, against the reality of storing bits with capacitors and wires, ie. analog devices. Clearly a leaky abstraction. The denser you pack those capacitors and wires, the more leaky.
Agree. The extreme secrecy of DRAM manufacturers about the innards of their chips puts an additional obstacles in the way of memory controllers (MCs) implementing efficient Rowhammer defences. In particular, if the MC doesn't know which addresses are corresponding to neighbouring rows, how can an MC know with certainty that any concrete row is being attacked? (And, to the best of my knowledge, DRAM manufacturers don't give away this information.)
The possibility of flipping bits in DRAM in a Rowhammer like fashion, was known in the DRAM industry since at least the 1990s (sorry, no reference handy), and Rowhammer-like access was used in DRAM quality testing.
As silicon density increased, the issue became more urgent.
Have you ever seen any even moderately detailed specification of what the DRAM manufacturers do in this regard? I have not, and I looked. I am deeply sceptical ....
I don't believe that Rowhammer mitigations happen inside the DRAM chips themselves, I think that they are being put into the memory controller that talks to DRAM. Since DRAMs with built-in Rowhammer defences would have to spend transistors on this defence, those transistors would be 'wasted' in situations where Rowhammer is not part of the attacker model.
[1] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, G. E. Hinton, Adaptive mixtures of local experts. (1991)
[2] M. I. Jordan, R. A. Jacobs, Hierarchical mixtures of experts and the EM algorithm. (1993)
[3] L. Xu, M. Jordan, G. E. Hinton, An alternative model for mixtures of experts. (1994)
[4] S. Waterhouse, D. MacKay, A. Robinson, Bayesian methods for mixtures of experts. (1995)
[5] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, J. Dean, Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. (2017)