FineWeb2: Adapting Pre-Training Data Processing to Every Languagearxiv.org7 points·by hynky·l’année dernière·0 comments
hynky·il y a 6 mois·discussHF recently released a blog about their pre-training dataset from PDFs.Alongside that they also shared a graph how much PDFs they were able to fetch by date, and the old internet seems to be mostly dead.
hynky·il y a 6 mois·discussHuggingFace recently released a report on their FinePDFs dataset, with this saddening report on state of current availability of old internet.
Alongside that they also shared a graph how much PDFs they were able to fetch by date, and the old internet seems to be mostly dead.