I hope/fear this HRM model is going to be merged with MoE very soon. Given the huge economic pressure to develop powerful LLMs I think this can be done in just a month.
The paper seems to only study problems like sudoku solving, and not question answering or other applications of LLMs. Furthermore they omit a section for future applications or fusion with current LLMs.
I think anyone working in this field can envision their applications, but the details to have a MoE with an HRM model could be their next paper.
I only skimmed the paper and I am not an expert, sure other will/can explain why they don't discuss such a new structure. Anyway, my post is just blissful ignorance over the complexity involved and the impossible task to predict change.
Edit: A more general idea is that Mixture of Expert is related to cluster of concepts and now we would have to consider a cluster of concepts related by the time they take to be grasped, so in a sense the model would have in latent space an estimation of the depth, number of layers, and time required for each concept, just like we adapt our reading style for a dense math book different to a newspaper short story.
This piece sounds like a second order LLM, the first-order one generates text and the second-order one filters and optimizes it. But, in fact, this work is done by a human being.
The human reviewer asks and wants you to proof-read your output before submission. It claims being able to detect any AI slop. I wonder whether this is true, and if so, for how long. Maybe it will be replaced by a GAN LLM, and then the loop will be closed.
Thanks for the April reference. In the post [1] there are just 11 comments and also it seems that the intended audience could maybe two dozen people!, not very encouraging to create a J in Common Lisp.
Sorry, but I don't get the meaning of your phrasing. I think that to use AI you must be very explicit and clear about what you want to design, and if Lisp provides some advantages one should define accurately the specific tool to use and when, how, and why.
I recall Norvig mentioning that other computer languages have taken many ideas from Lisp, those languages are also in the new civilization. Just to give an example: destructing-bind, apply and others are now done in javascript with a shorter syntax, and javascript without macros has excellent speed.
Thanks, it seems you are right. The random-sample project seems so short that I thought it was a one day project and that the other would be also short projects.
If a case doesn't match, then speeding up the remaining 0.1% is not going to change much the overall speed. Hence, a non optimized algorithm maybe enough. But when speed and speed variance is critical, then optimization is a must.
Perhaps instead of coding many small one day projects one could program one day projects that compose with each other. For example, I was thinking about developing a library that implements a version of J in Common Lisp (but I think fuel is lacking) so that, for example, the one day project named random-sample could be just:
randomSample =: 4 :'(x ? #y){y' NB. can't repeat.
randomSample =: 4 :'(? x # #y){y' can repeat.
So that, in many cases, one day projects could be reduced to one or two lines definitions (for those that know J that is the caveat).
Or the 8 solutions in a 2x12 matrix: