How are you finding rust for audio development? I have a background in pro audio, and both for the audio and GPU render threads, I used a lot of arena allocators and ring buffers. Do you find yourself fighting with rust's strict semantics?
I guess it's possible that Russia and/or China delivered some hardware to the Iranians. Doesn't seem far fetched given the low international support for this "excursion". Both countries benefit from a US quagmire.
The irony of an IP scraper on an absolutely breathtaking, epic scale getting its secret sauce "scraped" - because the whole app is vibe coded (and the vibe coders appear to be oblivious to things like code obfuscation cuz move fast!)...
And so now the copy cats can ofc claim this is totally not a copy at all, it's actually Opus. No license violation, no siree!
It's fucking hilarious is what it is, it's just too much.
The irony of an IP scraper on an absolutely breathtaking, epic scale getting its secret sauce "scraped" - because the whole app is vibe coded (and the vibe coders appear to be oblivious to things like code obfuscation cuz move fast!)...
And so now the copy cats can ofc claim this is totally not a copy at all, it's actually Opus. No license violation, no siree!
It's fucking hilarious is what it is, it's just too much.
Interesting article; I've always been fascinated and intimidated by FPGA programming - it's one of the few remaining "dark arts" of software engineering.
> VHDL’s delta cycle algorithm is its crown jewel. It gives you built-in determinism. Let us treasure it - Verilog doesn’t have anything like it. At the same time, you will agree with me that there is nothing too complicated about the concept.
> The deal was always simple: search engines had permission to crawl sites because they were going to be sending users to those sites. If they're hitting your site half a million times for every one user they send to your site, all they're giving you is higher costs.
You made me laugh!...Bjarne indeed can't be accused of being a modest man. And by some accounts, he's quite a political animal.
But in fairness, when was D&E first published? Argued for auto there, long before their acceptance. Argued for implicit template instantiation - thank god the "everything-must-be-explicit" curmudgeons were vanquished there, too.
He's got a pretty good batting average - certainly better than Herb Sutter.
Can you share what aspects of the design you (and Stroustroup) aren't happy with? Stroustroup has a tendency of being proven right, with 1-3 decade lag.
I'll have a stab at this. I'll start with an attempt at justifying the remark that an agent which is a good coder will be good at other tasks.
1. Coding is, as a technical endeavour, relatively difficult (similarly for mathematics). So a model which performs well on this task can be expected to easily handle also-technical-but-slightly-easier tasks, like understanding (musical) harmony theory or counterpoint -- for much the same reason that human programmers/mathematicians/scientist don't struggle to understand those "easier" theories.
2. Reinforcement learning augments a base models ability to excel in something else that's "difficult", namely to "look ahead" and plan multiple steps in advance. That's literally how the training algorithm works, generating multiple paths at once, and rewarding intermediate steps in those paths which succeed in attaining the goal. And that skill, too, is extremely useful in other domains. An AI agent which learns that to break a problem into sub-problems, and then tackle each in turn methodically -- it stands to reason that it can apply that to, say, a business plan.
Note: 1 & 2 are not independent, nor are frontier models' excellence in these domains magical: it ultimately boils down to the availability of massive datasets (in particular for coding) and totally objective metrics (in the case of mathematics: solved math problems). That's the key ingrediant for reinforcement learning to be so effective.
So: the skills are transferrable because they're difficult, and require lots of planning. That models are so good at them is a fluke, and in a parallel world where humans created git repo after git repo of business plans, it might be that which we lean on to teach a reinforcement learning algorithm how to "reason" and "plan".
Now let's turn our attention to the "synergies" aspect, which I agree with. Let's say your agentic model, which is already excellent at reasoning and planning, acquires a new or improved capability which allows it to search the domain space, calculate, etc. much better than before -- this capability can now bear upon the plan, or be factored into the plan. For example, the model might be able to say "I don't need to worry about this particular subproblem for now; I can rely on my "mathematica" capability to deal with it when I absolutely need."
Or to put it differently: monkeys, like humans, are able to use (rudimentary) tools. They'll take a rock, and use it to crack open a coconut (or whatever). But a human being, with far superior reasoning and planning abilities, takes that tool, and uses it to make an even better tool -- and the result after many iterations of this process is civilization as we know it, while monkeys are still stuck trying to crack open nuts with rocks.
In my experience, an indicator that my interlocutor is (possibly) lying/BSing, when challenged for an explanation for something they did, is that they provide a list of reasons. The person who's telling the truth just gives one.
This -- and obviously David Ng's article -- are absolutely fascinating pieces of work.
I have a few (very naive) questions:
There is a widespread intuition, encapsulated in the very terms "feed-forward networks" and "deep neural networks", that computation in such networks is akin to a circuit wired in series. My "observation" is that residual layers offer an "escape hatch" from this, allowing layers (or sets of layers), to operate in parallel (and of course, something in between).
So here are my dumb questions:
1. Is my intuition about residual networks, at least in principle, allowing for in parallel layers, correct? Or am I missing something fundamental? Let's say the intuition is correct -- is it possible to measure the degree to which a layer operates in series or in parallel?
2. The formula for residual layers (at least to my mind) reminds of an Ornstein-Ühlenbeck time series process. If so, can we measure the degree of mean-reversion of a/several layer(s)? For me, this makes intuitive sense -- the goal of avoiding vanishing gradients feels similar to the goal of stationarity in time series processes.
3. Let's take as an article of faith the central idea of a tripartite network: input->latentspace block => reasoning block => latentspace->output block. Ng's intuition iiuc is that the reasoning block, more or less, wired in series. Intuitively, it feels like that is what it ought to be (i.e., a chain of calculations), though I'll add -- again hand-wavingly -- that OP's efforts appear to cast doubt on this conjecture. Are the two "translation" blocks wired "more" in parallel, then?
4. So what both Ng and OP did was to "tape together" the ostensibly reasoning layers -- in different ways but that's essentially it. Another thing you could do is to treat the input and output translation blocks as fixed. You now train a totally new model on a much smaller corpus of training data, only instead of feeding the input directly to your new model you feed it translated training data (similarly, your targets are now the activations at the entrance to the reasoning->output block. Let's assume it's exactly the same architecture in the middle as the standard netowrk, only it's initialized to random weights as per usual. Surely you should be able to pre-train that 6 layer reasoning network much, much faster. Has anyone tried this?
5. Having thus partitioned a very deep architecture into three distinct parts, there's no reason why you can't experiment with making the reasoning block wider or narrower. Has anyone tried that?
6. Another fun idea is to map a given input through input block and read the pre-reasoning activations. You now let that vector be a random variable and do a random walk through reasoning input space, and use this to "augment" your corpus of training data. Reasonable idea or bullshit?
Please remember, I'm only just (and belatedly) trying to wrap my head around how transformer architectures work -- I'm still waiting for my copy of "Build a Large Language Model (from scratch)"! I hope these questions aren't totally daft!
I agree with the author that changes are needed to stymie the proliferation in malicious packages (almost certainly potentiated by the ubiquity of LLMs), but I wonder if this is the best way to go about it.
The author posits two scenarios allowing for bad actors to take control over packages to add malicious code: stealing the original maintainers credentials, and taking ownership over dormant packages.
I would guess the second is much more likely. At least, it's the only one I've witnessed myself (in Arch Linux - I now view AURs are somewhat risky). I acknowledge this is argument based on anecdote - but what isn't up for debate is that it's far easier to identity dormant packages with high fan-out than stealing someone's credentials.
The cool down approach addresses the symptom; it would be better to address the bad actors who are the cause: dormant packages, recently taken over by maintainers lacking prior reputations and with high (transitive) usage, should be viewed with high levels of suspicion by the package managers. We should assume the new maintainers are malicious, and require them to prove they are not.
From a language design viewpoint, I wonder whether the current crop of async implementations - eg 'def' -> 'async def', f(x) -> 'await f(x)', 'return 1' -> 'async return 1' are making a mistake analogous to the decades-long opposition to type deduction, ostensibly for "clarity".
Let me phrase this as a question to prevent "social" arguments against a homogenous syntax for coroutine: if coroutines were await-invoked using the same syntax as regular functions were invoked, would that create some unresolvable ambiguity for an important use-case (and similarly for the other syntactic forms)?
My gut feeling is that the answer is no. Consider what a C++ compiler does with an awaitable function (this may be specific to stackless coroutines): they are transformed into state machines, each state lowers into something akin to a regular function, local variables accessed across states are stored in a dynamic environment. Ok, but then isn't a regular function simply a coroutine whose lowered state machine has a single state? And that being so, why distinguish syntactically between the two?
It seems silly, to my mind, to do all this "work" (in the compiler) to create an abstraction which allows the programmer to forget whether 'f(x)' returns synchronously or asynchronously, only to introduce a syntax which requires them to know or look up which of the two it is. What does it matter which of the two it is, if the whole point of the exercise was to turn callback spaghetti into linear code (and allow to leverage automatic variable lifetimes aka RAII)?
The whole point is that this fits on an ESP32, which has wifi. We're not quite at the point where it makes sense to run the whole thing locally - if you do try it, it will need a fan, and be loud etc.
For my part, I installed Nanoclaw on my Arch derived OS (I love Arch!), and it worked fine until the next day some update decided to revert the power management settings, and now my glorious assistant is dead.
There's something to be said for a barebones OS. No bullshit, no updates.
Also, playing with hardware watchdog timers and GPIOs and DACs can be so much fun.
How are JVM startup times nowadays? Frankly the need to ship with a JVM (modulo graal admittedly) is a drawback, eg for CLI tools - which the author listed as a requirement. Go's static executable story, combined with it's competitive performance - without C++ or rust contortions - is a strong combo when your focus is on startup time and deployment simplicity. (If, otoh, I cared a lot about GC I'd definitely prefer Java - eg for non-fast algo trading.)
Java is a fine language, tech stack, and ecosystem, but I agree with the author and parent commenter that this is a sweet spot for Go. Their decision to use it makes a lot of sense.
How are you finding rust for audio development? I have a background in pro audio, and both for the audio and GPU render threads, I used a lot of arena allocators and ring buffers. Do you find yourself fighting with rust's strict semantics?