HackerTrans
TopNewTrendsCommentsPastAskShowJobs

wenhan_zhou

no profile record

Submissions

A Bitter Lesson for Memory

personal-website-3bed.onrender.com
4 points·by wenhan_zhou·21 giorni fa·3 comments

Show HN: Continual Learning with .md

github.com
34 points·by wenhan_zhou·3 mesi fa·34 comments

comments

wenhan_zhou
·21 giorni fa·discuss
Yep. Or even better, compact after a random number of turns. The model must then learn to preserve useful context at arbitrary context lengths.
wenhan_zhou
·21 giorni fa·discuss
If understanding emerges from pre-training, then perhaps memory is what emerges from post-training.
wenhan_zhou
·3 mesi fa·discuss
Currently working on a benchmark!
wenhan_zhou
·3 mesi fa·discuss
Ah, so you are effectively offloading the file exploration mechanism to the INDEX.md in the sub-directories rather than writing a complex prompt?
wenhan_zhou
·3 mesi fa·discuss
How does the agent intelligently synthesize information across different files?
wenhan_zhou
·3 mesi fa·discuss
In theory, yes. Although the privacy setting says otherwise.

But in the end, it doesn't really matter; it is public on GitHub, so anyone can use it.
wenhan_zhou
·3 mesi fa·discuss
I just read LLM Wiki in more detail. I have heard about it second-hand before this project. The "no-code" idea was inspired by Karpathy.

As I have understood it, in LLM Wiki, the human is very much in the loop in what gets written. In ReadMe, the human control is mostly on the policy (prompt) level, and it is done once, the agent then goes full autonomously afterwards.

After a quick skim of your project.

I have tried an embedding-based knowledge base as well, but it is a bit tricky to make the embedding match a user query. For example, "What happened?" is not at all similar to "Batman defeats Joker." You need to reformulate the query using an LLM, which is tricky given that the query is conditioned on the whole chat history. That's partly why I abandoned embedding-based methods.

But given that MCPTube already works on Gemini CLI, I could see it work natively without embeddings. Gemini is capable of reading video files natively. Worth a try?
wenhan_zhou
·3 mesi fa·discuss
Although I have been working on memory before, ReadMe is very fresh. The moment I saw it running, I published it. So, no continuous running nor LLM ablation studies.

Treat it as an MVP, would love to hear how your agent performs!
wenhan_zhou
·3 mesi fa·discuss
I think what's missing is a benchmark that measures how well the memories contribute to future interactions.
wenhan_zhou
·3 mesi fa·discuss
I don't remember such details, but as you suggest, it is a healthy kind of compression.

I address it through merging the lower-level memories into more abstracted ones through a temporal hierarchical filesystem. So, days -> months -> quarters -> years. Each time scale focuses on a more "useful" context since uncertain/contradictory information does not survive as it goes up in abstraction.

For example, A day-level memory might be: "The user learned how to divide 314 by 5 with long division on Jan 3rd 2017."

A year-level memory might be: "The user progressed significantly in mathematics during elementary school."

From the perspective of the LLM, it is easier to access the year-level memories because it requires fewer "cd" commands, and it only dives down into lower levels when necessary.
wenhan_zhou
·3 mesi fa·discuss
I see your point.

A removal mechanism is not (yet) implemented. But in principle, we could adjust the instructions in Update.md so that it does a minor "refactor" of the filesystem each day, then newer abstractions can form, while irrelevant gets pruned/edited. That's the beauty of the architecture, you define how the update can occur!

But if you do have a new memory (possibly contradicting an old one), is it really a good idea to prune/edit it?

If you are genuinely uncertain between choice A and B, then having them both exist in the memory archive might be a feature. The agent gets the possibility of seeing contradictory evidence on different dates, which communicates indecisiveness.
wenhan_zhou
·3 mesi fa·discuss
Minimalism is my design philosophy :-)

Good question. Since it is just an LLM reading files, it depends entirely on how fast it can call tools, so it depends on the token/s of the model.

Haven't done a formal benchmark, but from the vibes, it feels like a few seconds for GPT-5.4-high per query.

There is an implicit "caching" mechanism, so the more you use it, the smoother it will feel.
wenhan_zhou
·3 mesi fa·discuss
Yep. Markdown is the future :-)
wenhan_zhou
·3 mesi fa·discuss
Fair concern.

ReadMe does support loading memories mid-reasoning! It is simply an agent reading files.

Although GPT-5.4 currently likes to explore a lot upfront, and only then responds. But that is more of a model behaviour (adjustable through prompting) rather than an architectural limitation.
wenhan_zhou
·3 mesi fa·discuss
Context bloat is real, but the architecture has the potential to solve it.

You need clever naming for the filesystem and exploration policy in AGENTS.md. (not trivial!)

The benchmark is definitely the core bottleneck. I don't know any good benchmark for this, probably an open research question in itself.
wenhan_zhou
·3 mesi fa·discuss
The editability is surely an underrated advantage, both for the program itself and the memories it generated.

I think in terms of noise, it is less problematic here because not everything is being retrieved. The agent can selectively explore subsets of the tree (plus you can edit the exploration policy by yourself).

Since there is no context bloat, it is quite forgivable to just write things down.