ajtejankar

9 เดือนที่ผ่านมาdiscuss

Exactly! I don't think Skills is a new algorithm but it's definitely a new paradigm of organizing your prompt. Essentially, dynamic context assembling with stuff crossing user boundaries which. They even mention that they are working on skill sharing across teams in an organization. You can take this expand to global user base sharing things with each other in an agent.

ajtejankar

9 เดือนที่ผ่านมาdiscuss

I have been independently thinking about a lot of this for some time now. So this is so exciting for me. Concretizing _skills_ allows, as you said, a common pattern for people to rally around. Like you, I have been going dizzy about its possibilities, specially when you realize that a single agent can be modified with skills from all its users. Imagine an app with just enough backbone to support any kind of skill. From here, different groups of users can collaborate and share skills with each other to customize it exactly to their specific niche skills. You could design Reddit like community moderation techniques to decide which skills get accepted into the common repo, which ones to prioritize, how to filter the duplicates, etc.

ajtejankar

9 เดือนที่ผ่านมาdiscuss

IMO LoRAs are no different from context tokens. In fact, before LoRAs tuned prompt vectors were a popular adapter architecture. Conceptually, the only difference is that prompt adapters only interact with other tokens through the attention mechanism while LoRAs allow you to directly modify any linear layer in the model. Essentially, you can think of your KV cache as dynamically generated model weights. Moreover, I can't find the paper, but there is some evidence that in-context learning is powered by some version of gradient descent inside the model.

ajtejankar

9 เดือนที่ผ่านมาdiscuss

Yup! I fully agree. It also taps into the ability of LLMs to write code given good prompts. All you need is for the LLM to recognize that it needs something, fetch it into the context, and write exactly the code that is needed in the current combination of skill + previous context.

ajtejankar

9 เดือนที่ผ่านมาdiscuss

With so many code sandbox providers coming out I would go further than you say that this is almost a non-problem.

ajtejankar

3 ปีที่แล้วdiscuss

There is an 'old' paper https://arxiv.org/abs/1701.06538 but I believe there needs to be renewed effort in this direction.

ajtejankar

3 ปีที่แล้วdiscuss

Fixing the experts for a layer might not work since all experts fire almost with equal probability. There are small variations by topic but they are consistent enough to be captured with a simple linear classifier. I believe this happens due to the load balancer loss which forces the model to pick all experts with equal probability. However, what you're saying is a great direction for future MoE's. Can we train MoEs without load balancing so that it is possible to quantize/prune the non-relevant experts more aggressively? We haven't had any major open source MoE's because, as far as I know, they are not straightforward to train, but I expect this to change.

ajtejankar

3 ปีที่แล้วdiscuss

The plots show 2 dimensional projection of the 8 dimensional feature vector of each paragraph. So, x and y axis are linear combination of 8 different experts. Ideally, all of this should be in a single plot but there was a lot of overlap between different sub-categories and it was hard to see. So, I thought of separating them by their broad categories. Also, there are 32 layers in the model, each has 8 different experts, and 2 of them picked for each token.

ajtejankar

3 ปีที่แล้วdiscuss

The base model has 32 layers and there is a single linear layer for language modeling (going from embeddings to the vocabulary) that gets applied at the very end.

comments