We ran this experiment a few weeks ago, but Anthropic’s Mythos report dropped this week and thought this would be relevant to share now.
Surprisingly, we found that for repos already in the training data(pre cut off set), the contamination is at such a high level that even by just giving the model the file name and not file contents, the model is able to tell what is inside that file. Same for file paths. Just given a file name, the model is able to correctly guess the file path - this implies that models already know the structure of these repos and thus understand what to look for and where.
This ability drops sharply for unseen repos(post cutoff set) - raises the question how effectively it will hold for private repos with proprietary scaffoldings and programming patterns. Then the question worth asking is how much of Mythos's capability on well known codebases like Firefox and OpenBSD is genuine reasoning vs parametric familiarity with their structure?
Methodology caveat: modest sample (9-10 repos per group), treat numbers as directional - more experiments in progress....
Déjà Code: How LLMs Cheat on Repos They've Seen · HackerTrans
We ran this experiment a few weeks ago, but Anthropic’s Mythos report dropped this week and thought this would be relevant to share now.
Surprisingly, we found that for repos already in the training data(pre cut off set), the contamination is at such a high level that even by just giving the model the file name and not file contents, the model is able to tell what is inside that file. Same for file paths. Just given a file name, the model is able to correctly guess the file path - this implies that models already know the structure of these repos and thus understand what to look for and where.
This ability drops sharply for unseen repos(post cutoff set) - raises the question how effectively it will hold for private repos with proprietary scaffoldings and programming patterns. Then the question worth asking is how much of Mythos's capability on well known codebases like Firefox and OpenBSD is genuine reasoning vs parametric familiarity with their structure?
Methodology caveat: modest sample (9-10 repos per group), treat numbers as directional - more experiments in progress....