HackerTrans
TopNewTrendsCommentsPastAskShowJobs

andrewgross

no profile record

comments

andrewgross
·letztes Jahr·discuss
Super easy to get started, but lacking for larger datasets where you want to understand a bit more about predictions. You generally lose things like prediction probability (though this can be recovered if you chop the head off and just assign output logits to classes instead of tokens), repeatability across experiments, and the ability to tune the model by changing the data. You can still do fine tuning, though itll be more expensive and painfaul than a BERT model.

Still, you can go from 0 to ~mostly~ clean data in a few prompts and iterations, vs potentially a few hours with a fine tuning pipeline for BERT. They can actually work well in tandem to bootstrap some training data and then use them together to refine your classification.
andrewgross
·letztes Jahr·discuss
Got it. I’ll review the paper again for that portion. However, it still sounds like the end result is not VRAM savings but efficiently and speed improvements.
andrewgross
·letztes Jahr·discuss
Ahh got it, thanks for the pointer. I am surprised there is enough correlation there to allow an entire GPU to be specialized. I'll have to dig in to the paper again.
andrewgross
·letztes Jahr·discuss
Is there a concept of an expert that persists across layers? I thought each layer was essentially independent in terms of the "experts". I suppose you could look at what part of each layer was most likely to trigger together and segregate those by GPU though.

I could be very wrong on how experts work across layers though, I have only done a naive reading on it so far.
andrewgross
·letztes Jahr·discuss
> The beauty of the MOE model approach is that you can decompose the big model into a collection of smaller models that each know different, non-overlapping (at least fully) pieces of knowledge.

I was under the impression that this was not how MoE models work. They are not a collection of independent models, but instead a way of routing to a subset of active parameters at each layer. There is no "expert" that is loaded or unloaded per question. All of the weights are loaded in VRAM, its just a matter of which are actually loaded to the registers for calculation. As far as I could tell from the Deepseek v3/v2 papers, their MoE approach follows this instead of being an explicit collection of experts. If thats the case, theres no VRAM saving to be had using an MOE nor an ability to extract the weights of the expert to run locally (aside from distillation or similar).

If there is someone more versed on the construction of MoE architectures I would love some help understanding what I missed here.
andrewgross
·vor 2 Jahren·discuss
4 x 32GB for now. I need to investigate manual OC as EXPO doesn't work with all 4 slots populated. Another option is to try the 192GB 4x48GB Corsair kits at 5200.
andrewgross
·vor 2 Jahren·discuss
7950x3d w/ 128GB at stock timings (~3200MT/s?). Showed a high base but no increase with threads, need to investigate what is happening.

  1 54.7
  2 50.6
  3 49.6
  4 48.2
  5 47.9
  6 47.4
  7 47.1
  8 46.6
  9 46.5
  10 46.2
  11 46.1
  12 45.9
  13 45.8
  14 45.7
  15 45.7
  16 45.7
  17 45.7
  18 45.8
  19 45.9
  20 45.8
  21 45.8
  22 45.6
  23 45.6
  24 45.5
  25 45.5
  26 45.5
  27 45.5
  28 45.4
  29 45.4
  30 45.4
  31 45.4
  32 45.4