This means that the results (both of the task and of the learning by the student) are lower if the student uses an LLM first, but slightly improves if they use it second
Exactly what I thought of reading this, that chapter is genuinely one of the most affecting things I've ever read. The horror of it keeps growing as he continues to describe awful manual task after the other.
I've found good results from summarizing my documents using a large context model then embedding those summaries using a standard embedding model (e.g. e5)
This way I can tune what aspects of the doc I want to focus retrieval on, it's easier to determine when there are any data quality issues that need to be fixed, and the summaries have turned out to be useful for other use cases in the company.
I was trying to do this recently for Web page summarization. As said below the token sizes would end up over the context length, so I trimmed the html to fit just to see what would happen.
I found that the LLM was able to extract information, but it very commonly would start trying to continue the html blocks that had been left open in the trimmed input. Presumably this is due to instruction tuning on coding tasks
I'd love to figure out a way to do it though, it seems to me that there's a bunch of rich description of the website in the html
Interestingly he recently discussed how using LLMs tends to remove this desirable difficulty: https://davidepstein.substack.com/p/a-risk-of-cognitive-conv...
This means that the results (both of the task and of the learning by the student) are lower if the student uses an LLM first, but slightly improves if they use it second