Not very far off, the IBM 1130 was very much built around its punch-card reader.
I've written a backprop in Fortran IV for the 1130 and while it works, it was tedious.
Add ten more years, and the IBM 801 could have been a CPU architecture good enough to scale all the way to the present day without emulation, unlike the 360.
The first AI winter was largely triggered by Minsky in a book he published in 1969, which mathematically proved that single-layer perceptrons couldn't solve non-linear problems. Favorite quote: "Our intuitive judgment is that the extension [to multilayer systems] is sterile."
Yet we had the computational power to run backpropagation in the 1960s and small Transformers in the 1970s (I'm the author of both):
What was missing wasn't the raw processing power, but the ideas and algorithms themselves. Because funding and research were completely discouraged during the AI winter, neural networks research was left dormant and we lost two decades.
In the early days of machine learning (before the first AI winter), networks like this were often implemented and trained in hardware: https://en.wikipedia.org/wiki/ADALINE
That was the first thing that came to mind when I read "the smallest brain you can build". Nowadays, that "small brain" would likely be built on a breadboard using op-amps instead.
Scheme has a coherent and minimalist design, but its ecosystem and abstraction facilities feel too sparse for large applications.
When I started building a Lisp-based machine learning framework, Guile seemed like the right choice because it provides GOOPS and generic functions, yet I still ended up with a lot of boilerplate to compensate for the lack of a strong type system.
Scheme feels to me like C is to C++: not ergonomic for large-scale application development. Go is one of those languages that has both minimalism and productivity.
I was thinking of curated replay buffers, which would act like "dreams". To prevent collapse, the offline dataset would mix the new mid-term data with a baseline of anchor data (the original training distribution) so the model doesn't drift.
Also, we wouldn't train on the whole session. A separate critic module, like a reward model, would filter the KV cache to extract the high-value information, like a garbage collector before the LoRA.
That's just an idea though. Right now most research focuses on changing the architecture itself (TITAN, HOPE...) instead.
That's an idea I had a few months ago: after going through a compaction once the KV cache is nearing capacity, accumulate this knowledge into a dataset to fine-tune a LoRA during offline hours.
This would create a three-layer memory system:
- Stable long-term memory (initial base weights)
- Mid-term memory built from the compactions and replay buffers
- Short-term memory (KV cache)
Sleeping would just be a fancy term for consolidating and transferring information from one memory layer to another during offline hours. Maybe that's also what the brain does while sleeping.
Strictly speaking, this is very domain-specific and doesn't enable any performance that Triton couldn't already achieve (eliminating global memory round-trips via epilogue fusion is nothing new). The real takeaway is the design shift for LLM-driven codegen rather than handcrafted kernels.
LLMs are still bad at low-level hardware optimizations, but really good at high-level composition. Designing compiler abstractions with a restricted, composable API so an LLM can easily glue expert-written blocks together is a smart move. I suspect this will eventually become the norm for codegens as we move to agentic development.
There is ongoing research on neural cellular automata, as they seem to be a very efficient way to generate pretraining tokens: https://arxiv.org/html/2603.10055v1
Emacs Lisp is a descendant of PDP-10 MAClisp, which makes it one of the oldest Lisp dialects still actively maintained. Whether it's version 24.5 or 30.2 doesn't make much of a difference semantically.
Obvious slop still makes it to the front page of HN, and sometimes farms GitHub stars.
These posts also usually get all these glowing comments from users who clearly haven't checked the code. It's even worse when authors get busted and claim "Okay, Claude wrote it, but the design is mine" despite clearly not understanding the output themselves.
Unfortunately, that makes high-effort projects less visible. The SNR will probably keep getting worse until slop can be flagged on HN.
I wonder how they plan to implement that on decentralized social networks such as Nostr. I assume targeting big centralized networks such as X and Facebook is good enough.
Same everywhere: avalanches of AI garbage and intellectual dishonesty. People claiming "I wrote this", then a look at the code shows massive slop and an author with no clue about the topic.
More worrying, this trend is creeping to all domains:
"Nearly 75,000 tracks uploaded to Deezer are fully created using AI. That’s 44% of daily uploads, and more than 2 million per month. Back in June, the daily number was around 20,000."
flinux essentially had the architecture of WSL1, while CoLinux was more like WSL2 with a Linux kernel side-loaded.
Cygwin was technically the correct approach: native POSIX binaries on Windows rather than hacking in some foreign Linux plumbing. Since it was merely a lightweight DLL to link to (or a bunch of them), it also kept the cruft low without messing with ring 0.
However, it lacked the convenience of a CLI package manager back then, and I remember being hooked on CoLinux when I had to work on Windows.
Add ten more years, and the IBM 801 could have been a CPU architecture good enough to scale all the way to the present day without emulation, unlike the 360.