> The next time you ask the LLM for another endpoint with the same access rules, the model won't start from first principles. It'll start from the other four copies already sitting in your repo.
To be honest I'm not sure how true this is. I think it's more that there does seem to be quite a baked-in bias to repeat basic structures and not reuse (much less come up with) abstractions. So where that is the existing pattern it looks like it's keeping with that, when in reality it would often do that either way.
There have been many cases where I've started a piece of work by laying down very rigid abstractions and a few examples of using them, and I explicitly prompt to not only exclusively use the specific abstraction API but also copy the way I've used it. And the (frontier) LLM does neither, it just steams ahead re-implementing things from scratch from bottom up basic structures, partially and often totally ignoring the abstractions.
I don't know exactly why this should be the case but my naive suspicion is that there's just an awful lot of this type of stuff in the masses of training code and the weights just somehow 'know better' how to get results this way, rather than using your more novel abstractions/patterns.
One way I like to think about is that often abstraction is an automation for a task that doesn't need automating.
You hardly ever change the thing and if you do, changing it in two or three places 'manually' is really not a big deal.
Now changing something fairly often, that affects logic in 50+ places? Then it makes sense to automate with an abstraction so it all flows through the same lines of code.
I know I've personally spent way more time over the years debugging bad abstractions than changing things in a few places.
I think there's a strong chance this is a case of it creating a whole new market though.
There are people whose current behaviour/situations will happen to benefit from this, and that may be a niche, but seems like there's a really solid chance many more people actually will change their behaviour in response to this being available. That's how disruption happens.
To be honest I can easily see the default changing if the service is good enough. I mean it seems like you basically get most of what is good about wired plus a whole load of extra previously totally unavailable benefits. For a price, to be sure, but that'll come down.
It's a clever argument because if you question it, you're reminded of the entire history of technological development which is, guess what, exponential.
You're sometimes also dismissed as not understanding the concept of exponentials. This again is clever, as it's baked into the definition that if you don't see it happening, or can't imagine it happening, well that's precisely a tell you're living through an exponential!
All the reasons you might give can be countered with, essentially, "that problem that seems clear today will go away sooner than you can imagine and when it does you'll be on the back foot, so you'd better just assume it will go away and project/plan accordingly".
The trick is entirely that one cannot possibly deny the general power of exponential progress across all of technology, it's almost a law, but it doesn't work in the other direction - no particular local technology is owed exponential growth because of this general pattern. Sometimes things just cap out at merely 'useful' and don't improve much further, no matter how much you want to believe they won't, no matter how steep the progress curve (or, indeed, line) has been up to that point.
To this point the narrative of what these tools can do over these last 3 or 4 years has always been way ahead of the reality. Everyone who works with the tools knows this.
Not everyone wants it to be true, so some will not acknowledge it and will just keep pushing this year-ahead projection as ground truth today. Many (not all) of those people aren't builders, so they don't have to deal with present reality jarring up against this projection of what ought to be possible, they're safe just talking about what should hypothetically be possible and making plans around that that won't be executed for months to years anyway. This keeps the flywheel going, and in fairness, some of the reality has actually caught up in certain ways, so some of those plans will have to some degree worked out which spins the flywheel faster still.
In the end though I just keep thinking: it's been 4 years (as referenced in the post). A lot has happened, the tools are very cool and very useful for certain things. But when I put my head up and look around in the world, even just the software world, nothing's really changed in terms of actual outcomes, in terms of new things appearing or being built that didn't exist 4 years ago. Certainly nothing feels instinctively like it's improved much, subjectively.
Maybe this is what it feels like to be in the knee of a curve of an exponential, but it seems equally reasonable this is just a breakthrough that's kind of improving at a clip you'd expect it to for all the investment put in, but fundamentally is just a new tool that needs to be slowly commercialised in an economically rational way, as we gear up for the next breakthrough which may or may not be related. Who says it must just keep improving forever? This argument never made much sense to me.
well to be fair that's right now, I think the question is what about in 6 months, 12 months, 2 years?
Where do these improvement curves go? Does the gap close, do they intersect for practical purposes (factoring in cost etc)? Or is the local curve always just a translation of the hosted, lagging behind, or indeed does hosted just pull ahead?
Nobody knows, but it's a very open question I feel, and it certainly appears like the answer might quite reasonably be that yes they intersect on that kind of short-ish term time horizon.
on the other hand failing at it or pushing it to the edges until they figure out where if anywhere it would actually make sense (which is I believe the entire crux of their strategy) might equally reasonably be helping their brand.
I don't see strong evidence the average consumer is demanding 'AI features' in everything. I mean even amongst the technically inclined this is often bemoaned, anecdotally.
This won't happen in most cases because the valuable thing is largely the knowledge encoded in the software, which the buyers of the software don't have and don't want to have since they're focused on their own business.
There's also, of course, the not insignificant value in the software itself actually working, being operated, being updated when necessary, all of that. Again just extra hassle no business will want to shoulder when they can just buy something that does it for them.
I agree and I think the most concrete demonstration of this for me is actually the current air.
It's basically the idea of the pro of 10 years ago but realised flawlessly, and improved upon actually in most ways, for significantly cheaper.
To do this they had to become an execution machine at huge scale to own and solve every piece that held them back in that pro design, and they did, and it's honestly amazing how it all came together.
I think there's a ton of examples where this is true for lower level stuff like open source where you see the internals.
For commerical products it certainly exists too, for example in those cases where you know the product is built by one person or a small group of people who you absolutely know take extraordinary care to get all the details right, and it shows through as a really nice intangible feeling when you're using the product.
That (kind of rare to be honest) 'oh this is just really well done' feeling.
Seems roughly right, that does seem to be about the boost in the most well-suited cases where you essentially know exactly how to solve the problem, the problem won't change much, and it's truly a matter of just churning out the implementation.
In that case precisely prompting, doing the review & nudge loop, can be a pretty nice (nice, still not game changing) speed boost over literally typing out the code to match the design in your head.
The less optimistic view though is that most things you build aren't like that. Even if they seem like it first. These things get booked as a nice speed boost, but you'll only find out much later they weren't.
A confounding factor is that it seems like many people not in the detail of building software do seem to think of most to all things are like that, even before AI assisted coding. Not much need to say more - see the entire history of the 'agile' movement for evidence of this.
And because most things aren't like that, I actually struggle to see fundamentally how more than 20-40% will ever be achieved (short of the ever-present deus ex machina of AGI argument), simply because the generation is already really good for these types of things. So since things like this aren't going to increase in overall proportion of things to be done, I don't see where the overall extra gains come from by models improving at this point.
I think unfortunately it's not about what seems obvious, or even what seems more likely, but about what seems retrospectively justifiable regardless of outcome.
The incentive structure of this type of decision is 'absolutely under no circumstances existentially mess up'. Ostensibly with respect to the organisation, but in actual reality much more so with respect to the individual(s) involved in the decision.
If everyone else is doing something that kind of obviously makes no sense, and you decide to break from the crowd by instead doing what does make sense, then there's a pretty solid chance of gaining a temporary edge while reality resolves the truth. But those gains probably won't matter all that much for the organisation, or indeed your position within it. It's a solid chance of an unimportant gain.
However on the other hand, there's a tail risk that something very unexpected happens and the thing everyone's doing that makes no sense actually turns out to make sense - sometimes even for entirely unpredictable incidental reasons - and then, well, you're in trouble. Not necessarily 'you' the organisation.. they'll likely be able to catch up and it won't matter that much. But for 'you' personally, the decision maker, it's very much not good.
As a bonus, in the much more likely scenario that the thing that makes no sense turns out to indeed make no sense, you're in the same boat as everyone else, there's no relative loss, and most importantly you don't stick out as someone who did something as risky as to go against the prevailing, albeit pretty clearly nonsensical, sentiment.
So basically, game theory tells you pretty quickly to just go with the thing that makes no sense if you're optimising for some (weighted) cross of what's best for the organisation and yourself as the decision maker.
I was just thinking how it'd be great if there were newer, modern things like this that had sprung up in response to newer technologies.
I guess it's one downside of dematerialisation with digital tech - I can't think of a single thing that would make sense. Everyone's got their own virtual portal to all the new technologies that come out, there's not much to look at out in the world.
Maybe as more progress happens in physical 'world of atoms' type things we'll see a bit of this come back.
I think weirdly, ads embedded in AI search responses actually maybe do have a chance at being helpful (as long as it's clear from the context of the question that I may be willing to pay for a solution) just because they could potentially be quite well matched to the specific thing I want, or if they're not quite as well matched but offer other benefits, explain the difference.
At the moment search ads aren't very helpful because you have neither of those things. You always get them for any type of query, and when you do get them you don't know if the thing being shown will exactly solve your problem, or only approximately, and the work is much more on you to find that out by reading the product's marketing pages further.
If all that could be done for you up front, reasonably honestly, then I could see it being useful. I mean to be sure, in some small percentage of searches I really am looking to buy something and really do want to be usefully, honestly pitched on available options.
interesting thing about the scoring effect narrowing the types of writing that generate the higher average ratings.
I guess also there's something fundamental about next token prediction that is going to narrow in on 'stable' ratings. Let's say for the 4/5 rating you have a wide range of styles, there's also this question of then the style of 4/5 isn't going to be cohesive, so which one do you pick?
Any output that mixes them up is going to get rated lower in later testing, and then if you have somehow to pick one, you've got to stick with it, which seems like it would be complicated to put into next word prediction weights. You'd need a different branch of weights for each style or something.
So not only does the rating process itself push towards a generic style, but the model 'prefers' training on this generic style, i.e. gets to better-seeming results faster.
it's almost certainly not true yet but at some point there might be an equilibrium reached of speed Vs quality (and let's not forget, cost) where it's true for most of what you do.
Perhaps you'd still turn to hosted models for the hardest tasks, but most tasks go local. It does seem like that would make demand go down significantly.
Of course that's all predicated on model advances plateauing, or at least getting increasingly more expensive for incremental improvements, such that local open source models can catch up on that speed/quality/cost curve. But there is a fair amount of evidence that's happening. The models are still getting noticably better, but relative improvement does seem to be slowing, and cost is seemingly only going up.
I think this instinct is intrinsic, and comes from really caring about detail and wanting to fully understand it and own it.
That's what drives it, and I don't really think the extrinsic things about the way you learned (while helpful) have that much bearing on it. It comes from you and you should take credit for it.
I think if you were learning today you'd probably find have the same feeling and do just fine because of it.
> Especially with the advent of AI, it feels like the time I had left to learn and actually build something has run out.
No, this is wrong.
Get your head down. Learn fundamentals. Practice and develop real skills. Ignore anyone saying this is irrelevant now. Let them talk, keep learning and building stuff, while using the new tools.
Give it a few years and you'll find the narrative has changed again, meanwhile you've got a few more years of experience under your belt. Avoid the noise and just focus on building.
I think a tell that many of these deals likely aren't real and are basically just PR is the numbers are super round and digestable.
That's a clear signal that little analysis has gone into the numbers and, most generously, there's nothing but the shape of a deal the details of which will be ironed out and adjusted in practice.
I get that the amounts of funding and capital being sat on for the respective parties are collossal and lead to rounding that doesn't make sense from the point of view of an individual any more (what's a few million at this scale, just round up to nearest 10, etc) but deal sizes of literally round numbers of 100s start to stretch credibility on whether any real analysis was involved.
In fact it'd be a ridiculous coincidence if it had been. They're the kind of figures where you'd recheck your calculations to check it's right as it seems too perfectly round.
a bit of an aside but what's amazing is that Docker's recent beta VM for Mac (I think released a couple of months ago now) has dramatically improved the performance you get out of your CPU.
Using a macbook air, even a recent one, before this Docker was definitely usable but noticably slower. Probably still worth it but a noticable tradeoff using it as a dev machine Vs a pro. Now that tradeoff has basically gone away.
Working on https://withdocket.com -- it's a system for active note taking in regular meetings like 1-1s.
I have a blog at https://davidnicholaswilliams.com
Email: my HN handle at google's email service. Put (HN) in the subject line to tell me you came from here!