The "redundacy" experiment is very interesting! Strongly agree, we just need to do something better than "dumping your entire corpus into a very large context window", maybe using this table-of-contents methods would be very useful?
Yeah, I agree it’s not something new, since humans also do this kind of retrieval. It’s just a way to generate a table of contents for an LLM. I’m wondering, when LLMs become stronger, will we still need vector-based retrieval? Or will we need a retrieval method that’s more like how humans do it?