Interesting project. Do you think manual memory management help understand computational graph lifecycle better, or does it distract from backprop itself?
btw, I went down the micrograd path with numpy-primitives all the way to building a PyTorch clone that can pre-train and post-train LLMs (https://github.com/workofart/ml-by-hand). My learning focus was on the math/calculus <-> high-level APIs, instead of efficiency. I'm glad to see more people tackling this problem from different angles.
I'm continuing to expand my own deep learning library [1] built with numpy-primitives to support LLM post-training techniques like supervised fine-tuning (SFT) and reinforcement learning with GRPO. It's a good learning experience to work without all the high-level abstractions to "build a wheel" and "use that wheel to build a car".
I'm also looking into coding harness self-improvement [2]. An inner LLM (raw LLM request) + harness solves coding tasks, an outer agent like Claude or Codex that proposes harness changes. I experimented with many things in the past few months that made me realize this self-improvement thing that everyone is talking about is just an experiment design problem. I wrote about it here [3]. I'm continuing to improve the infra around the self-improvement loop, to increase signal-to-noise ratio per experiment. I'm also generalizing the infra to expand beyond terminal bench tasks and to collect some data across different models (harness-bound vs model-bound).
I don't think the market will swallow the stock offerings until we see early signs of GDP growth attributable to these entities. But until then, I think the cost is higher than the benefit, which "The dead economy theory" essay covered it well [0]
I recently wanted to see whether an AI agent could self-improve a harness to solve terminal bench tasks. It’s possible for an AI agent to propose a meaningful one-time change to the harness, but after experimenting with this for a couple of weeks, I think the continuous self-improvement is mostly an experiment-systems problem. The system needs a way to decide what kind of improvements can safely compound.
Turns out there's a lot of parallels to coding-agent customization (e.g. SKILLS.md etc..) too.
I wrote my experience of building such system here, including the successful and failure attempts during the process, and how I approached the self-improvement loop. It's not intended as a benchmark claim but more of a systems/research writeup.
I'm continuing to expand my own deep learning library [1] (PyTorch-clone built with Python and Numpy) to support LLM post-training techniques like supervised fine-tuning (SFT) [2] and reinforcement learning with GRPO [3] . It's a good learning experience to work without all the high-level abstractions to "build a wheel" and "use that wheel to build a car". Post-training results are still cooking, since training on my MacBookPro is quite slow with "unoptimized PyTorch" :)
Interesting to see more demand shaping mechanisms applied to LLM inference. Even though the "batch processing" feature is already available. I guess this "promotion" is to test the hypothesis of sliding along the spectrum towards more "real-time" demand shaping.
Thanks for sharing! It's inspiring to see more people "reinventing for insight" in the age of AI. This reminds me of my similar previous project a year ago when I built an entire PyTorch-style machine learning library [1] from scratch, using nothing but Python and NumPy. I started with a tiny autograd engine, then gradually created layer modules, optimizers, data loaders etc... I simply wanted to learn machine learning from first principles. Along the way I attempted to reproduce classical convnets [2] all the way to a toy GPT-2 [3] using the library I built. It definitely helped me understand how machine learning worked underneath the hood without all the fancy abstractions that PyTorch/TensorFlow provides. I eventually wrote a blog post [4] of this journey.
Love the educational value of this "nano-sized" project. This reminded me of the from-scratch project I created to learn about deep learning libraries, neural networks all the way to LLMs like GPT-2 using just Numpy and Python [1]. Learning is done by "re-inventing the wheel" yourself, one step at a time :)
btw, I went down the micrograd path with numpy-primitives all the way to building a PyTorch clone that can pre-train and post-train LLMs (https://github.com/workofart/ml-by-hand). My learning focus was on the math/calculus <-> high-level APIs, instead of efficiency. I'm glad to see more people tackling this problem from different angles.