Seems like for such requests it's necessary to get some proof of work: require a meeting where for every artifact they sent you to review, they briefly explain the gist and point out the motivation for creating the artifact.
Agreed. It seems a core issue underlying these prompt injection attacks is a failure to properly scope the agent's permissions. In this case, depending on what exactly the agent is supposed to actually do, this might be defining a separate workflow agent per repo, or a workflow agent with broader repo access but configured to only be triggered by users on an allow list (still compatible with developing in the open, still allows outsiders to open public issues, but takes into account the different trust to be placed in each). And likely many more options when one properly thinks about it.
But that requires:
1. the technical ability for such fine-grained scoping / permissions
2. actually taking the time to think about what you want to achieve with the agent and what the smallest set of permissions / capabilities is for it to achieve it
Regarding 1., I think this will come, we're still in the wild west phase of agent usage. It'll be interesting to see which abstraction(s) will turn out to be the best interface for humans designing agents (minimize friction for finding and defining scope and permissions) and to limit agent capabilities (again finding the best trade-off between level of detail possible for defining capabilities and the ease of use of actually doing it).
Regarding 2., well, that's still the core problem that's always prevented the construction of high quality software, isn't it? Taking the time to properly think it out,and then taking the time to properly implement it. Which goes counter to the "move fast and break things" approach of people throwing agents at everything.
I agree with the basic premise: that when using LLMs, they will tend towards some mean when resolving ambiguity.
There's a very interesting opportunity here for a deeper investigation.
- What does "regression to the mean" actually mean in practice when the LLM is conditioned on a possibly large amount of context?
- How does this perceived regression to the mean affect the result in different applications? When implementing code, it may show up as keeping it simple, hence easily understandable, "nonclever". When writing documentation, it may show up as simple language, short sentences, etc. supporting the intent of communicating with little friction to a broad audience. When brainstorming product ideas, it may show up as regurgitating old and boring ideas, but dressed in fancy language and affirmations that hide the shallowness of the content.
- What can be done to alter this behavior? Now that temperature doesn't seem to be a parameter anymore in new models, how can we steer creativity of the model?
- If the model's creativity is fundamentally limited, is there a way we can use it to support us in the expression of our creativity, leveraging the different strengths of humans and LLMs in a way that the result transcends the limits of either?
Unfortunately, I don't see the article doing that. And, while I know pointing out LLM-isms is often a cheap shot these days, I feel compelled to point out that this article is full of what I perceived as LLM-ism, quite ironic given the premise and the statement ("written off-distribution · on purpose").
E.g.
> Trained on the past, it answers in the past tense of thought. Not what is true. What is typical.
> We converge — not on what is right, but on what is average.
> Not the answer it was sure of — the one it would not stop correcting
I'm sure there's plenty of writing in the above style to be found on the Internet, and hence having been trained on by the LLM. I'm also not a fan of this style, and in particular I'd say it's rarely or never found in scientific / technical writing meant to convey understanding rather than sell or hype. So here it's IMO more of a style mismatch.
Not sure about taking it down to the level of consciousness, but makes sense regarding the sense of self, the conceptual experiencer, the perceived center of experience. It agrees well with the observation I have made again and again they my sense of self is much stronger when I'm around people, and stronger still when I'm in a context where I don't know people and/or am uncertain in social rules.
This can be as immediate as dancing in a club, and closing my eyes I feel open, free, still, the body just flowing, then opening my eyes and feeling the cage of categorization of the world, relating my self to other people as a major function, coming right back.
Also being alone in nature for me makes the sense of self drop. Without intention, spending even just a few hours alone in a forest seems to quiet down the part modeling my self in relation to the world so much. There's no need for it there. I'm not a person in a forest; I become the trees, the birds, the rustling of the leaves, the sun shining through the canopy.
Even if we know it's a machine we're interacting with, since the instructions we give are so similar in form to how we interact with people, I'd be very surprised if those interactions wouldn't affect how we communicate in general. After all, we are creatures of habit to a much larger degree than most would like to admit.
So I'm in the same boat: I'd much rather "look silly" being polite / kind to a machine, than have the most effective way of using it decay the kindness I'm habituated to express towards people.
> It reminds me of people who take drugs and get "revelations" but then are not particularly over represented in the group of successful people for all of their deep insights.
This depends on where you're looking for "successful" people.
I generally agree with you - of those people who might report "revelations" through hallucinogenic drugs, the majority may misinterpret their drug-induced experience and hence be more confused / lost than before.
On the other hand, it can still be true that among those who eventually do have genuine spiritual insight, having used hallucinogenic substances is overrepresented compared to the general population.
Quoting from [1], where the author tried to find spiritually advanced individuals:
> Approximately 52% of
participants had used hallucinogenic drugs at
some point; none reported these as the trigger
that led to PNSE.
PNSE = Persistent Non-Symbolic Experience.
My point is: while there are certainly people who go way overboard with the LLM stuff, that is not at odds with skillful use of LLMs being overrepresented in successful people.
I see now that you didn't make that point, but I already typed this all out and I'm gonna leave it.
> We don't hire juniors and throw them boilerplate and tiny bugs while expecting them to learn along the way ad hoc through some pair programming and the occasional deep end.
Is that generally the case though? I'm about two years into my first job in the industry and that's exactly my experience, and certainly frustrating...
> There are ways to robustly clean this up analytically but it is largely beyond the capabilities of current tech stacks.
Can you expand on that? Even just conceptually that sounds really hard, how would you know whether you're measuring genuine (unexpected) changes in the environment rather than the result of (possibly sophisticated and coordinated) deliberate manipulation?
I agree. Don't have a citation now, but I remember reading that this was a copyright problem. They wanted to name it "Linux Subsystem for Windows", but apparently the Linux foundation does not allow unaffiliated projects to have a name beginning with "Linux", or something like that.
Maybe you are saying the same thing, but couldn't that be explained better by those people being afraid to be made obsolete? Or at least, afraid if having to retrain?
Not sure about the underlying reason, but I use Windows for work and the only program I've encountered in the past two years with this behavior is the Eclipse IDE. Everything else deals very well with rescaling and docking / undocking to 4k displays.
> It's a terrible long term lubricant (because it's designed to evaporate, it actually concentrates gunk and grime).
I recently read that WD40 isn't actually a lubricant but a lubricant remover. So as you write you'd use it to remove gunk but then follow it up with an actual lubricant.
On the last two bottles of WD40 I came across (im Germany) I checked the back and it indeed said that it's not a lubricant but a lubricant remover.
(Disclaimer: can't read the article past the intro where it does call it a lubricant...)
Two years ago I did some cleaning up and finally sorted out the gaming PC from my youth. I believe I bought it around 2007. Ran some old AMD dual core (may have been an Athlon 64 4400), still had an HDD. Installed on it was Windows Vista, which wasn't exactly a crowd favorite. So as I went to backup the final remnants of those gaming days I was flabbergasted by the snappiness of the explorer. Folders just opened instantly! So snappy, it was actually fun just navigating through all the folders. I had been expecting this PC to run at snail's pace, yet the windows experience was much better than on my desktop PC built in 2021 running Windows 10 on an NVMe drive. I have no idea how that is possible, but since then with every interaction with modern Windows there's just this tiny tinge of sadness...
My guess would be yes, it's cultural. I'm not Indian but spent about 5 months there. Overall my impression was that people act much more on direct feedback.
It would be typical to do the first thing that comes to mind, then see what happens. No negative feedback? Done, move on. Negative feedback? Try the next best thing that makes the negative feed back go away.
People will not wonder whether they might bother you. Just start talking. Maybe try to sell you something. That's often annoying. But also just be curious, or offer tea. You react annoyed and tell them to go away? They most likely will and not think anything bad of it. You engage them? They will continue. Most likely won't take "hints" or whatever subtle non-verbal communication a Westerner uses.
I found it quite exhausting in the beginning, it feels like constantly having to defend myself when I want to be left alone. But after I started understanding this mode and becoming more firm in my boundaries, I started to find it quite nice for everyday interactions. Much less guessing involved, just be direct.
Professionally I haven't worked much with Indians, but my expectation would be that it's necessary to be more active in ensuring that things are in track. Ask them to reflect back to you what the stated goal is. Ask them for what you think are obvious implications from the stated goal to ensure they're not just repeating the words. Check work in progress more often.