(don't get me wrong. love C. but in an innocent sort of way, like a teenager quite unaware of betrayals, heartbreak, love triangles, or UB, UsB, and IDB..)
Great, we can get authoritative answers. (I'm trying to understand the ML space and have mostly done readings, not an expert.)
I am assuming you can have n LoRA fine-tunings, say each specializing in one aspect of a coherent task, with n summers, running in parallel, and then combine them at the end? Or more generally, does LoRA enable a sort of modularizing around a core (un-merged) model?
And curious if you ever tried merging 2 or more fine-tunings and then testing the resultant single model (merge all) against the original tests to check retention?
The hit seems to be in energy/cpu not time since the W0 computation is in parallel with the BAx. (My assumption based on the latency claims in paper.) So an issue in edge deployments (battery life, etc.).
> you are stuck with that one model on that device
Upfront I have 0 clue on the actual numbers, but from a purely software architecture pov [in unmerged setup], having that W0 forward process once with n distinct BAx paths (for distinct fine tunings!) would address that, no?
[p.s. say an application that takes as input A/V+Txt, runs that through an Ensemble LoRA (ELoRA™ /g) which each participant contributing its own BAx finetuing processing, sharing the single pre-trained W0.]
"The other direction, as exemplified by prefix tuning (Li & Liang, 2021), faces a different challenge. We observe that prefix tuning is difficult to optimize and that its performance changes non-monotonically in trainable parameters, confirming similar observations in the original paper. More fundamentally, reserving a part of the sequence length for adaptation necessarily reduces the sequence length available to process a downstream task, which we suspect makes tuning the prompt less performant compared to other methods."
This is key imo: "More fundamentally, reserving a part of the sequence length for adaptation necessarily reduces the sequence length available to process a downstream task".
The LoRA paper’s ‘problem statement’ makes a compelling case for practical benefits of the approach. Specifically, no added latency, no serial processing bottlenecks, shared baseline model, compact time/space requirements. How does dreambooth stack up in this regard?
“Aghajanyan et al. (2020) shows that the pre-trained language models have a low “instrisic dimension” and can still learn efficiently despite a random projection to a smaller subspace.”
Would be great to have an informed practitioner comment (sota) on why we opt for random projection. Is the actual ‘intrinsic’ vector space uncomputable? Too slow to find?
Photo is an interesting word. It's meaning is clarified by other words, such as photorealism, photofinish. These words will (strictly) lose their meaning if photograph simply means image captured and processed by a device.
Curiously and revealingly, the political word photo-op stands alone in this photo- parade of words in the age of photo-imaginings. The universe does indeed have a sense of humor.
Possibly effective doodling in the design space of systems? I personally hacked at untold number of experiments with different approaches, and was doodling like mad over at least a decade. I have bookshelves full of notebooks covered with object graphs, system sketches, etc. It's not just coding.
I somewhat take issue with this. (3+ decades of very hands on (read: coding) architecting, including some learning experiences in orbit. I've been coding code-doodling since teenage years.)
A lot of what non-coding architects traditionally brought to the table has been taken over by experts designing OSS protocols, data formats, etc. Take things like data frames that are now exploding in ML space. Seniors today, agreed, should be able to integrate (sub)systems and that may in fact be enough. But a competent systems architect (who have never touched code) should be able to also define data formats, patterns of movement of data between sub-systems (for say optimal performance), the actual computing platform considerations, etc. Also, sometimes when being too close to code, things degenerate to debates about tools, etc.
Naturally my points here gain more validity as system size (or its open-ness requirements) increase.
That trivializes our profession and hard earned experience simply because a degree is not a requirement. It is ridiculous to say drag'n'droping something is halfway to being a systems designer. And the anecdote about "experience at facebook" was such nonsense. "I worked at facebook doing something other than distributed systems, and geez, I learned nothing about distributed systems". "QED!"
p.s. I missed this ad hom bit (no, not the issue here)
> If you have an issue with people leveling up, good luck to you.
"You can pass system design interviews even if you’ve never designed distributed systems before. If you have copied files between machines with drag-and-drop, you are halfway there. If you implemented clients or servers or have opened network connections, you’ve got this. This guide will teach you the most important 20% of information that will appear 80% of the time in system design interviews. By the end of this guide you won’t be an expert, but you’ll be well on your way to being a better engineer and a much better interview candidate."
A better engineer, no less! Wow, this must be some magical guide. I have to read it now. Also what does this say about our industry. "You want to be a surgeon but don't have the experience? Ever cut a tomato? You're halfway there. If you've ever made a sandwich, you've got this. Nurse!"
The perennial issue for competent organizations as they scale is the quality of the middle management. Alex didn't write anything about hierarchies of molds (because it most certainly isn't moldy all the way to the top and never was). His pithy slides also don't discuss hiring practices at Google for managers. [We all know how they grill us poor doers.] He also failed to mention our dear friend Peter of the principle fame.
p(goal) = f(p(planners), p(managers), p(doers)) is a more realistic equation. What that f() looks like depends less on organizational structure than on the quality of the workers that mediate planning and building.
The subject at hand affects the entirety of society, not just hn members. Even today surely you must have very intelligent and capable friends or people you know that are preoccupied with their own domain and it is an unreasonable position to expect them to keep up with software and hardware and cellular/networking and be able to ascertain with high confidence that X is a safe and secure choice.
This issue can not be decided based on what is appropriate for a niche segment of society. We have a variety of options to serve the general society -- state/control, state/licensing, state/regulation, private/corporate, public/ngo, public/community, personal/ai, personal/expertise. These are not mutually exclusive options and we should discuss the possible configurations, optimal for general societal welfare, rather than denying that the problem exists.
There is certainly that, but in fairness GP is correct that I am saying that at some point tech is going to outstrip even the best of us (maybe it's just the dumb me projecting).
One possibility that comes to mind is that we could possibly rely on trustworthy AI to help us understand the implications of using X in full. So if not walled gardens maybe private landscape architects or AI gardeners.
But one of the two will be required for digital life.
At some point in the near future we are all going to be in the same shoe as GP's mother: confronted by tech we simply no longer understand. Even the latest generation is merely fluent in using these devices. The subset that will keep up will be even smaller than today, imo. Almost all others will choose the safe options, specially when our lives are even more critically intertwined with our devices.
Even today it is not ok to be careless with what you put on your machine, but in a couple of decades, it could be a true disaster, specially if public services are fully tied to your digital identity. You would not want your digital self to be entertaining any random* program as guest.
(don't get me wrong. love C. but in an innocent sort of way, like a teenager quite unaware of betrayals, heartbreak, love triangles, or UB, UsB, and IDB..)