nstart

3,935 karmajoined il y a 12 ans

[ my public key: https://keybase.io/kiriappeee; my proof: https://keybase.io/kiriappeee/sigs/96k3PXwMa2aveYHJWNbYvfGtJsqHVZ36pFm-4iN3HPM ]

Blog at https://adnanissadeen.com/blog

comments

il y a 4 joursdiscuss

Because, as the gp pointed out, if the cost is least to the labs, then why not reap the benefits too?

Hypothetical. Assume you can in fact point agents at a tool and say "replicate it. Make no mistakes". You then have software being instantly copy-able.

Assume these agents can then be pointed to a customer feedback board in perpetuity and they autonomously upgrade the software over time. They analyze usage patterns and behave like PMs figuring out what to prune and what to build. Then the maintenance part of the stack also goes to zero.

Over time, the highest margin competitiveness will go to the distributor of the tokens. Aka the AI model makers.

In a world like that (which the frontier labs claim is within a year or two of happening) it feels like it's only a matter of time before they opt to own the entire stack down to the consumer apps. Kind of like Amazon deciding they want to knock off products doing well and then favour their own product over the original seller.

My guess is that if the capability arrives the only reason the frontier labs don't move to own the entire stack immediately is because of optics. Boil the frog instead.

il y a 24 joursdiscuss

Right. But this is why I want to know the prompt. My hunch is that the author knew this story. But likely prompted Fable without hinting at it. And if so, the fact that Fable defaulted to the story of Samson shows that while it can impressively extend that over so many "scrolls", it also could only generate the idea based on what it had gobbled up. I'm thinking of this because given to another human, I doubt they'd only ever go for Samson's story by default.

il y a 24 joursdiscuss

I tend not to focus on that future too much. I used to do so long ago. For example, how could Facebook possibly justify their losses while asking for such a big valuation? Same for Uber. Same for any number of big companies. And it turns out that growth in the future is impossible to predict accurately. Shopify is a good example where at the time the addressable market of online stores was tiny. But it turned out that Shopify created its own market which is huge today. Technology improvements have a way of creating new markets which far surpass today's total addressable market. Factor in currency depreciation and whatnot and sometimes, futures that looked impossible turn out to be possible.

Not saying anyone is wrong in pointing at the buildouts for AI and questioning its feasibility. Just making the argument for why I personally only look at operational costs and revenue because it's the only real-ish value I can look at and judge if a business can grow sustainably.

As a counter point, the red flag to all of this is R&D costs growing for each model release. If that continues and revenue cannot outstrip it, then these companies have a problem and it'll probably be that just 1-2 frontier labs can survive this once the dust settles.

il y a 25 joursdiscuss

I'm a little confused here. Cost of revenue is lower than revenue. That's good. R&D is the main contributor to losses here and this seems normal in an industry like this. For OpenAI specifically, I think this is problematic. They were the first movers but despite the large R&D they've lost so much ground to Anthropic despite Anthropic seemingly gifting them with weird PR self owns. But if we were to extrapolate this to the industry as a whole, this seems more positive than negative. Am I reading this incorrectly? Unless there's an assumption that R&D costs have to forever go up in order to increase revenue, I feel like this shows that the AI industry is actually on a path to profitability in the long term.

Whether it can physically be as all encompassing as it makes itself out to be or whether it will just be healthily profitable remains to be seen. Kind of like how Uber went from "We'll autonomously drive the world" to "Look, we deliver food, goods, and people to locations and we figured out how to do that in a way that makes profits. Also, ads".

il y a 28 joursdiscuss

Tbh, there really needs to be some legal precedent set that makes model distillation a legal activity. If the model makers can rip everyone else's work and launder information as if it's their own without giving credit back to the original creators, I don't see why it should be illegal to distill the models. It's the same thing the frontier model makers are doing to IP everywhere else.

le mois dernierdiscuss

Desperate to know what the prompt for the poem is. The idea of it felt familiar so I went down the rabbit hole and found: 14 years ago, a poem on reddit [https://www.reddit.com/r/RedditDayOf/comments/tjjw2/may_12_a...] . Nowhere near the length of the one the author shared but the same idea.

> This is from "The Cyberiad", a collection of science-fiction fairy tales by Polish author Stanislaw Lem ... In one of the stories, a robot constructor named Trurl creates a machine that writes poetry. A jealous rival named Klapaucian challenges the machine to compose "...a poem about a haircut! But lofty, noble, tragic, timeless, full of love, treachery, retribution, quiet heroism and in the face of certain doom! Six lines, cleverly rhymed, and every word beginning with the letter s!!"

And the computer responds with:

"Seduced, shaggy Samson snored.

She scissored short. Sorely shorn,

Soon shackled slave, Samson sighed.

Silently scheming,

Sightlessly seeking

Some savage, spectacular suicide"

The author had to be referencing this moment in their challenge to Fable/Mythos. I'm curious to know what their exact prompt was.

le mois dernierdiscuss

Also, I submit Ed zitron's the era of the business idiot.

https://www.wheresyoured.at/the-era-of-the-business-idiot/

Lots of little claims I disagree with there but the overall thesis has felt prescient these days

il y a 2 moisdiscuss

That is a very poor comparison. Firstly, you're ignoring that behind the mass manufactured stuff, there's a lot of abused labour. People do protest that. Because of how money works, it's also impossible to avoid it.

Second, machinery that automated work isn't remotely the same. Engineers have built and refined the machines without having to go and inspect every new work that has been created by artisans each time. Creative people who have practiced the art of designing clothes and shoes stitch together and build prototypes. Entire machinery is built as an independent path away from how artisans build furniture.

There is a parallel though for how LLMs, in order to improve, gobble up all new work produced by people and never give attribution back. We see it when someone does a unique physical product design and starts selling it only for some 2 bit shop elsewhere to try and copy and sell a cheap knockoff version. The original person does all the hard work of prototyping and testing and the 2 bit shop which has access to more machinery resources buys a couple, copies it with less quality, makes a few changes, sells it, and probably outspends the original person on ad revenue too.

No, GenAI doesn't produce the exact same work as what they ingest. But style does get reproduced. And style is such a difficult problem to solve. Studio Ghibli didn't craft its signature style by accident. People prototyped and worked hard on how to design it, how to solve the problems unique to the design, created rules for it, and then painstakingly made the stories that were best told through that style. Only for the AI companies to pop out some bastardized version of it every time someone says "make my picture anime". No attribution given. No love. No homage. Just an encouragement for hordes of people to claim how easy it is without ever understanding the thought that actually went into it.

So no. It's not hypocrisy. It's recognition of these machines being information and creative laundering factories. They take and take and never give back any value that they could never create or improve on on their own. Those last words being key

il y a 2 moisdiscuss

Agreed. What's frustrating is that we have models for how sandboxing can work and instead of investing efforts into nailing that experience, the OS providers are prone to turning it into a monetization/lock in layer instead. My VLC and VS Code should have an OS native way of being limited to particular functionality. But when the OS providers implement the sandbox, they center it around an App Store and restrictions on only apps that have been notarized where said notorization costs money or a requires a subscription. And then they remove the ability to do things which their own native apps can do and set tighter controlling rules on what APIs apps can ever have access to.

When all I wanted was for VLC or similar to run in a sandbox by default where a plug-in I install can't do anything to my system or access the internet by default because the software itself is restricted to just the files I'm using and that's it.

il y a 2 moisdiscuss

Yep. I like to sometimes think of Agents as a slow but tireless unofficial API glue between completely disparate elements. I dislike the approach the companies took in laundering information through them. But their ability to work through clerical work that no one should be doing ideally has been incredible for me. Of course there's nuance to that last sentence. Someone needs to have an eye on certain things in domains like finance. But it's up to companies to be smart here and ask how AI can augment that process rather than trying to get rid of it. For example, collecting all relevant context and presenting it on demand for a suspicious transaction flag would eliminate what could be days of inter departmental wrangling in a normal work process.

My only side note of sadness here is that companies are more likely to implement such stuff in a haphazard way rather than anything actually thoughtful.

il y a 2 moisdiscuss

Thats the thing though. The reduction of agents to Chatbots in a fancy dress doesn't make sense. Whether there's much of a moat to the models agentic approaches is a different question. But the idea of reframing questions and results in a back and forth between itself while holding on to context and all the laundered knowledge it has (no I'm never letting go of the lack of ethics in its knowledge acquisition) is an impressive feat. To say it isn't doing much is to liken someone doing any kind of thinking as doing nothing much. I'm not saying LLMs are sentient or intelligent in a human sense. But their synthetic intelligence does have capabilities that are impressive and they are capable of reducing so much busy work for me personally. Those ai agents that aren't apparently doing much can help me narrow down my reading about prior art in security implementations super quickly by going site by site, categorizing, locating the correct page in docs, or a Reddit discussion, extracting a relevant paragraph, sourcing it, and putting it down for me. The idea that this isn't much is reductive. That would have been 2 hours of my energy previously and instead I pick up the completed work, visit the links I asked the agent to get for me, do my reading in a pre planned structured way, and complete my work (I always try to be respectful of source material instead of attributing to the LLM). That is very useful behaviour and I think we thumb our nose at it to our own detriment.

il y a 2 moisdiscuss

I've been subscribed to ed for a long time. I commend his foundational ideas like what he laid out in "The Era of the Business Idiot" or "The Rot Economy". My recommendation line for him to anyone else is "if nothing else, he'll leave you with something to chew on for a while to come".

My issue with Ed is that he doesn't have the ability to draw the line. In the pursuit of making a point he goes so dogmatic that he is willing to make harsh statements that go beyond number backed predictions. Like in his piece "AI is really weird" he states about agents, "Probably the weirdest thing about this entire era is how nobody wants to talk about the fact that AI isn’t actually doing very much, and that AI agents are just chatbots plugged into an API.". That's a massive stretch to make. Just because he has a claim that the business doesn't make sense, he doesn't get to claim that agents are not capable of doing very real work. His assessment of cowork was "a chatbot that deleted every single one of a guy’s photos when he asked it to organize his wife’s desktop.". These statements damage his credibility and make it too easy to dismiss his writing as a rant of an angry man.

il y a 2 moisdiscuss

This is a misleading headline. It makes it seem like another supply chain attack where some good plug-in was taken over and used to deliver malware. Thats not the case here. Victims are invited to collaborate on a synced vault which comes preloaded with a non official plug-in that delivers the rat. Very very different story

il y a 4 moisdiscuss

Before software, there were accountants. It was The qualification to have.

Today accountants are still needed. But it's a commodified job. And you start at the absolute bottom of the bottom rungs and slave it out till you can separate yourself and take on a role on a path to CFO or some respectable level of seniority.

I'm oversimplifying here but that is sufficient to show A path forward for software engineers imo. In this parallel, most of us will become AI drivers. We'll go work in large companies but we'll also go work in a back room department of small to medium businesses, piloting AI on a bottom of the rung salary. Some folks will take on specialisms and gain certifications in difficult areas (similar to ACCA). Or maybe ultra competitive areas like how it is in actuarial science. Those few will eventually separate themselves and lead departments of software engineers (soon to be known as AI pilots). Others will embed in research and advance state of art that eventually is commoditized by AI. Those people will either be paid mega bucks or will be some poor academia based researcher.

The vast majority? Overworked drones having to be ready to stumble to their AI agent's interface when their boss calls them at 10 PM saying the directors want to see a feature setup for the meeting tomorrow.

il y a 4 moisdiscuss

Good spot! That is the product working as intended though. The background doesn't exist except as an asset that replaces the green screen. The tool is meant to replace the green screen without the need for manual rotoscoping. Even in a traditional process, the distortion needs to be done by VFX as a separate process. To do that though, they still need the green screen keyed out and this tool does that.

il y a 4 moisdiscuss

Yup. That's correct. And I understand that. I was looking at the changes to yarn.lock that got reintroduced. I couldn't figure out what was happening. It turns out that not only was it force pushed, but GitHub also retains the old commit information even if it's been "deleted".

I still don't quite understand what GitHub is doing to allow someone to say that dependabot coauthored a spoofed commit. This isn't the commit message itself I'm talking about. It's the GitHub interface that officially recognizes this as a dependabot co authored commit. My hunch is that the malicious author squashed two commits, the original good commit to yarn.lock and a malicious change to package.json, and that somehow maintains the dependabot authorship instead of reassigning it fully to the squash-er.

il y a 4 moisdiscuss

I don't quite understand how this is working tbh. I looked at one of the affected repos, ironically named "reworm".

The malicious code was introduced in this commit - https://github.com/pedronauck/reworm/commit/d50cd8c8966893c6...

It says coauthored by dependabot and refers to a PR opened in 2020 (https://github.com/pedronauck/reworm/pull/28).

That PR itself was merged in 2020 here - https://github.com/pedronauck/reworm/commit/df8c1803c519f599...

But the commit with the worm (d50cd8c), re-introduces the same change from df8c180 to the file `yarn.lock`.

And when you look at the history of yarn.lock inside of github, all references to the original version bump (df8c180) are gone...? In fact if you look at the overall commit history, the clean df8c180 commit does not exist.

I'm struggling to understand what kind of shenanigans happened here exactly.

il y a 4 moisdiscuss

Yea. It's a pretty lol-sob future when I think about it. I imagine the agent frameworks eventually getting trusted actors and RBAC like features. Users end up in "confirm this action permanently/temporarily" loops. But then someone gets their account compromised and it gets used to send messages to folks who trust them. Or even worse, the attacker silently adds themselves to a trusted list and quietly spends months exfiltrating data without being noticed.

We'll probably also have some sub agent inspecting what the main agent is doing and it'll be told to reach out to the owner if it spots suspicious exfiltration like behaviour. Until someone figures out how to poison that too.

The innovation factor of this tech while cool, drives me absolutely nuts with its non deterministic behaviour.

il y a 4 moisdiscuss

Every communication point (including whatsapp, telegram, etc) is turning into a potential RCE now. And because the agents want to behave in an end to end integrated manner, even sandboxes are less meaningful since data exfiltration is practically a feature at this point.

All those years of security training trying to get folks to double check senders, and to beware of what you share and what you click, and now we have to redo it for agents.

il y a 4 moisdiscuss

That's really cool. Do you have any write-ups I can checkout? I'm still new to this area of offensive sec so would love to learn from folks who've been in the thick of it.