(don't get me wrong. love C. but in an innocent sort of way, like a teenager quite unaware of betrayals, heartbreak, love triangles, or UB, UsB, and IDB..)
Great, we can get authoritative answers. (I'm trying to understand the ML space and have mostly done readings, not an expert.)
I am assuming you can have n LoRA fine-tunings, say each specializing in one aspect of a coherent task, with n summers, running in parallel, and then combine them at the end? Or more generally, does LoRA enable a sort of modularizing around a core (un-merged) model?
And curious if you ever tried merging 2 or more fine-tunings and then testing the resultant single model (merge all) against the original tests to check retention?
The hit seems to be in energy/cpu not time since the W0 computation is in parallel with the BAx. (My assumption based on the latency claims in paper.) So an issue in edge deployments (battery life, etc.).
> you are stuck with that one model on that device
Upfront I have 0 clue on the actual numbers, but from a purely software architecture pov [in unmerged setup], having that W0 forward process once with n distinct BAx paths (for distinct fine tunings!) would address that, no?
[p.s. say an application that takes as input A/V+Txt, runs that through an Ensemble LoRA (ELoRA™ /g) which each participant contributing its own BAx finetuing processing, sharing the single pre-trained W0.]
"The other direction, as exemplified by prefix tuning (Li & Liang, 2021), faces a different challenge. We observe that prefix tuning is difficult to optimize and that its performance changes non-monotonically in trainable parameters, confirming similar observations in the original paper. More fundamentally, reserving a part of the sequence length for adaptation necessarily reduces the sequence length available to process a downstream task, which we suspect makes tuning the prompt less performant compared to other methods."
This is key imo: "More fundamentally, reserving a part of the sequence length for adaptation necessarily reduces the sequence length available to process a downstream task".
The LoRA paper’s ‘problem statement’ makes a compelling case for practical benefits of the approach. Specifically, no added latency, no serial processing bottlenecks, shared baseline model, compact time/space requirements. How does dreambooth stack up in this regard?
“Aghajanyan et al. (2020) shows that the pre-trained language models have a low “instrisic dimension” and can still learn efficiently despite a random projection to a smaller subspace.”
Would be great to have an informed practitioner comment (sota) on why we opt for random projection. Is the actual ‘intrinsic’ vector space uncomputable? Too slow to find?
Photo is an interesting word. It's meaning is clarified by other words, such as photorealism, photofinish. These words will (strictly) lose their meaning if photograph simply means image captured and processed by a device.
Curiously and revealingly, the political word photo-op stands alone in this photo- parade of words in the age of photo-imaginings. The universe does indeed have a sense of humor.
Possibly effective doodling in the design space of systems? I personally hacked at untold number of experiments with different approaches, and was doodling like mad over at least a decade. I have bookshelves full of notebooks covered with object graphs, system sketches, etc. It's not just coding.
I somewhat take issue with this. (3+ decades of very hands on (read: coding) architecting, including some learning experiences in orbit. I've been coding code-doodling since teenage years.)
A lot of what non-coding architects traditionally brought to the table has been taken over by experts designing OSS protocols, data formats, etc. Take things like data frames that are now exploding in ML space. Seniors today, agreed, should be able to integrate (sub)systems and that may in fact be enough. But a competent systems architect (who have never touched code) should be able to also define data formats, patterns of movement of data between sub-systems (for say optimal performance), the actual computing platform considerations, etc. Also, sometimes when being too close to code, things degenerate to debates about tools, etc.
Naturally my points here gain more validity as system size (or its open-ness requirements) increase.
That trivializes our profession and hard earned experience simply because a degree is not a requirement. It is ridiculous to say drag'n'droping something is halfway to being a systems designer. And the anecdote about "experience at facebook" was such nonsense. "I worked at facebook doing something other than distributed systems, and geez, I learned nothing about distributed systems". "QED!"
p.s. I missed this ad hom bit (no, not the issue here)
> If you have an issue with people leveling up, good luck to you.
"You can pass system design interviews even if you’ve never designed distributed systems before. If you have copied files between machines with drag-and-drop, you are halfway there. If you implemented clients or servers or have opened network connections, you’ve got this. This guide will teach you the most important 20% of information that will appear 80% of the time in system design interviews. By the end of this guide you won’t be an expert, but you’ll be well on your way to being a better engineer and a much better interview candidate."
A better engineer, no less! Wow, this must be some magical guide. I have to read it now. Also what does this say about our industry. "You want to be a surgeon but don't have the experience? Ever cut a tomato? You're halfway there. If you've ever made a sandwich, you've got this. Nurse!"
The perennial issue for competent organizations as they scale is the quality of the middle management. Alex didn't write anything about hierarchies of molds (because it most certainly isn't moldy all the way to the top and never was). His pithy slides also don't discuss hiring practices at Google for managers. [We all know how they grill us poor doers.] He also failed to mention our dear friend Peter of the principle fame.
p(goal) = f(p(planners), p(managers), p(doers)) is a more realistic equation. What that f() looks like depends less on organizational structure than on the quality of the workers that mediate planning and building.
This interview is, among other very interesting threads of various ideas, like a capsule course in writing fiction from a master. It is well worth reading as it is not simply about the story, but rather how he put together a story, constructed layers and characters, and something like sketching out the mechanics of the dynamic between author, text, and readers:
FH: You see, and so we turn the whole thing whirling backward through the story. There was another thing there, in the pacing of the story, very slow at the beginning. It’s a coital rhythm all the way through the story.
WM: It’s a what?
FH: Coital rhythm.
WM: OK.
FH: Very slow pace, increasing all the way through, and when you get to the ending of it, I chopped it at a non breaking point, so that the person reading the story skids out of the story, trailing bits of it with him. On this I know I was successful, because people come to me and say they want more and…
It's a great read. Of course issues of power, structures of power, "the voice", and all things Dune are there as well.
The problem, in all cases, remains the long standing oligarch families and aristocratically rooted institutions, and their captive “public service” institutions, some of which are global in scope. Not wishing to engender a flaming thread, I will simply state that certain aspects of “institutional capture” are very much du jour topics of global interest and impact.
A global reset of “free markets” via a ‘day zero of capital accumulations’ could provide a solution. Many of the established capital hordes are legacies of activities that are now understood to be anti-social at best, and predatory at the extreme.
Coupled with this, we need pedagogical guidance to inform the new generations who are not to manor born. Almost none of the new blood born to middle or lower classes are educated in the necessities of generational wealth preservation and applications of wealth towards affecting societal outcomes. At best, we have children of Marxists and pseudo-Marxists railing against “Capital” without understanding the dynamics of societal power based on multi-generational societal networks, which transcend mere capital.
Primary sources working against such a program are precisely the “entertainment” complexes owned stock, lock, and barrel by informed and purposive societal networks, which at this point in human history have fully transcended ethnic and national boundaries, and clearly aim for stupefying the masses. There is a reason you have been treated to 2 decades of Marvel comics in films.
Is the OP claiming that he/uber built systems that have no discernible architecture?
Confusing methodology with architecture is not helpful. Even the simplest system has 'architecture'. Now this can be arrived at via: happenstance methodology, or expression of internalized well known types, or via a more formal process.
(don't get me wrong. love C. but in an innocent sort of way, like a teenager quite unaware of betrayals, heartbreak, love triangles, or UB, UsB, and IDB..)