The removal of “noindex” from the Internet Archive, and associated risks(old.reddit.com)
old.reddit.com
The removal of “noindex” from the Internet Archive, and associated risks
https://old.reddit.com/r/DataHoarder/comments/156s7di/the_removal_of_noindex_from_the_internet_archive/
3 comments
Like one of the commenters in that thread said, this sounds like they were using the noidex feature to use the IA as a personal private backup, and thus abusing it, and ruined it for everyone else. The IA is great as a personal public backup. (For example, I've deliberately submitted copies of certain OSS projects I've worked on to the Wayback Machine.)
The IA can always defer to putting a census of noindexed items as a json item though if they want to stay true to their motto of "universal access to all knowledge". For every invalid reasons there are always a valid reason for retaining such features. Some people might not want to see their works easily scraped by AI companies and so on which then profit off their backs, among all other sensitive use cases. If they do away with the self noindex option, the search results will one day become very clogged if you want to look for a refined result.
Permanent.org which is more geared toward the personal private backup use cases were only formally released in 2020. Before it there was only IA which could really provide digital preservation for all swathes of people. Even then I occasionally saw that Permanent.org cross-pollinates their datasets into the IA.
Contrary to the claims that I'm not looping in with them about noindex, in fact they had already been keenly aware of my (and probably others) use cases for a while. Before then I'd sometimes even contacted their staff members for help if I for some reason cannot no-index the items alone. Their tone changed slightly to say that "IA is a library" around the time of the bookseller's lawsuit but coincidentally because all the items I wanted to be archived, had been so, I was able to promise them to not upload any private items, at least I could discuss with Mr. Kahle about the issue. I did sent an email to the latter afterwards, but perhaps because he's heavily distracted by things like the lawsuits, it was eventually let slide and forgotten, until now.
I actually conversed with the founder somewhere before this year where I floated the possibility of the IA becoming an all-out archive and digital vault, more than just a library, in which he seemed to be welcoming with.
Edit: At one point IA even endorsed/recommended the use of noindex as an option to hide bulk-uploaded files from search results to avoid clogging it.
https://archive.org/post/1086430/should-i-archive-this-ever-...
Permanent.org which is more geared toward the personal private backup use cases were only formally released in 2020. Before it there was only IA which could really provide digital preservation for all swathes of people. Even then I occasionally saw that Permanent.org cross-pollinates their datasets into the IA.
Contrary to the claims that I'm not looping in with them about noindex, in fact they had already been keenly aware of my (and probably others) use cases for a while. Before then I'd sometimes even contacted their staff members for help if I for some reason cannot no-index the items alone. Their tone changed slightly to say that "IA is a library" around the time of the bookseller's lawsuit but coincidentally because all the items I wanted to be archived, had been so, I was able to promise them to not upload any private items, at least I could discuss with Mr. Kahle about the issue. I did sent an email to the latter afterwards, but perhaps because he's heavily distracted by things like the lawsuits, it was eventually let slide and forgotten, until now.
I actually conversed with the founder somewhere before this year where I floated the possibility of the IA becoming an all-out archive and digital vault, more than just a library, in which he seemed to be welcoming with.
Edit: At one point IA even endorsed/recommended the use of noindex as an option to hide bulk-uploaded files from search results to avoid clogging it.
https://archive.org/post/1086430/should-i-archive-this-ever-...
Full excerpt:
Surely those who are long-time users of the Internet Archive must have known about the parameter "noindex" in items which makes it hidden from the internal search engine, although remains on their servers. This is essential cases where which the uploader had uploaded content that is controversial in certain jurisdictions but doesn't violate US law in any way and they fear that making it publicly in search engines will invite unnecessary hassles and liabilities to the Internet Archive. Modi BBC documentary might be one tenuous example.
Besides, some people might be fine that their data will be studied by archivists in the far future, say a hundred years or more, however paradoxically in terms of near future and present they are not really okay for privacy/security reasons and prefer to see it noindexed, as opposed to the dichotomy that either an item is fully public, or doesn't exist at all, which in one way or more defeats the purpose of an "archive".
Here's one of the plausible example. A political activist or so want their memories to be archived for posterity becuase they think it will be helpful for far future archivists/researchers who study their career et al. However they prefer to see it hidden but archived because right now bad actors could use that info to doxx or harass them.
There was even an informal policy by Internet Archive to noindex some YouTube videos out of copyright-related reasons, per this.
However, they had remove the function from casual usage about May or June, perhaps after the mass-scraping DDOS incident, and unhide most of the items that are used to be noindexed. Therefore I had emailed the Internet Archive to ask for an explanation and to get them reverse the decision which I think is hare-brained.
After a few going forth and backs, they finally come with this reply which I feel is totally ludicrous:
There is no bug or mistake in removing no-index settings for many Internet Archive items in the Community collection.
At no point was the Archive contacted to arrange a situation of no-indexing (or Darking) items with an intention of later release; the no-index setting was not documented for this use, and represented a security hole that was closed. Tens of thousands of items were found, being used for encrypted files hidden from the search engine, and represented a major problem, so many items have been removed or set noindex quickly.
A number of people have contacted us explaining situations where items might need to be made no-indexed, in a collection for later or timed release for example, but they've done it with communication and discussing their needs, not just uploading files under disposable accounts and then assuming the archive would keep them un-accessible in perpetuity. In some cases their requests have gotten arrangements so that community items that were noindex are noindex again, in separate collections.
A situation can theoretically exist where the original uploader can e-mail us from their e-mail address and discuss arrangements, but you've indicated you intentionally obfuscated your location and have disposed your addresses. If you're able to gain access again, you can mail through those addresses.
An additional situation is you can e-mail [email protected] if you want to report items at the archive (by identifier) that you believe might need to be removed from the archive; we receive a number of these requests throughout the months and respond according to policy.
It is as if enshittification which had plagued Reddit just not long ago, has now made its landfall to the Internet Archive. The removal of "noindex" had in my two cents, destroyed the chance to attain delicate balance between preservation and privacy, hence meaning increased vulnerability to privacy and copyright related risks.
Perhaps, if the enshittification proves irreversible, there is Texas-based permanent.org which may one day become a successor of the Internet Archive.
Here's a pseudocode for them in case they come to their senses:
noindex items if:
(
items-noindexed-by-user-in-the-past = true;
OR items-noindexed-by-IA-in-the-past = true);
AND (
items-get-reindexed-voluntarily-by-USER-before-May-2023 = false;
OR
items-get-reindexed-voluntarily-by-IA-before-May-2023 = false;
)
Bonus:
Bit bloody arrogant isn't it, assuming you'll be of historical interest in a hundred years time and that IA ought to start keeping records of you now for that.
Bonus:
Some here said that hidden files are bad because they can be abused, but that doesn't mean that legitimate uses of the function can be handwaved away with a wand, just like how a coin has two sides. Thus it appears that there's a catch-22 situation here.
However, there's always the possibility of making a census in a separate area (whether internal or not, as a json file or something) of these hidden files if the Internet Archive wants them to remain accessible in the very long run. Furthermore it's a time sink for the IA employees if all the noindex requests are set to go through them manually every time; I bet they don't appreciate having to review these and that hence distracting them from more important pursuits such as scanning books.
send them an email and loop them in
In fact, they had been briefed, both implicitly and explicitly on the relevant use case scenarios through email recently and previously. They at one point stated that Internet Archive is a library, even though there are differences between the terms "library" and "archive".
Put it simply, think about the Library of Congress and the NARA. Public access to the latter's collections is normally harder than the former. IA had already chose to die on the "archive" hill when they used that term to brand themselves, and it's not a stretch to describe it as enshittification making a landfall when they tried to deviate from that core purpose as embodied by the term.
Finally as a disclaimer, under the penalty of perjury I am in no way had condoned and supported CSA behaviors, let alone upload those heinous items to the IA.
Bonus:
You're doing something nefarious. Otherwise you'd be more forthcoming about what you're uploading. The only way you're going to be able to recruit people to your cause is if you're more transparent.
Bonus:
Please keep the snarky BS elsewhere. All I've mainly uploaded to the IA are personal memories and creations, dating as far back as childhood. With the present context. one additional benefits of noindex function is that greedy companies can't easily scrape it and profit it on the back.
More than that, I admit that I engaged in some political activism for Ukraine and Hong Kong in the past and present, the latter which had placed me under the radar of their notorious National Security Law.
You're being entitled. Just because they used the word 'archive' it doesn't mean they're obliged to follow your interpretation of what that means.
If you want data preserved, pay for it. Don't expect IA to do it for you.
It's a non-profit, period, which are sustained through donations of any kind. After all it walks and behaves as if it's a poor man's archive all along. So much for "Universal Access to All Knowledge" if you force everyone to pay in order to upload their files. Without question you all will turn sour if that logic is applied to the Wayback Machine.
Edit:
IA is not your personal cloud backup. They're not obliged to host it for you, privately or publicly. Being a political activist is neither here nor there.
Hmm. I must say that the conception as quoted above, is no longer unfamiliar because those were normally used to stonewall criticisms against unpopular changes by tech companies or platforms, most recently by spez and his supporters during the API protests. That being said, you should be careful next time when you're entertaining it at all.
By the way, Reddit had now made photos in posts pretty much unarchivable.
Bonus:
It's strange to see that long-term thinking has been unexpectedly reviled here. I suppose that many veteran archivists who would've fully understood such idea, have joined others to leave Reddit in the wake of unpopular API changes.
To understand what "long-term thinking" is, think of this: Let's say someone is a small-time actor. They might make a breakthrough one day in Hollywood, or others so much that they become a household name. Their supporters, detractors and others will certainly see overarching interests into seeing their works and deeds preserved, for scrutiny, commemoration, and any imaginable purposes.
The hypothetical actor is a believer in archiving and digital preservation before famous, so they put a lot of items in the IA for years. While they are certainly okay with all their items being discovered and publicized by digital archaeologists a hundred years later or more, they would rather prefer that some of their items being noindexed, undisturbed, perhaps for a hundred years or indefinitely, like the artifacts in the pyramids, since premature exposure would mean TMZ and other unwelcome harassments.
As can be seen in this video, the IA founder is in acceptance with the general idea of long-term thinking and preservation. It's no longer "arrogant" and "unfeasible" to expect that personal legacies be preserved at any level and degree according to the user's wishes. For now I'll end this long comment with this Twitter thread by the Long Now Foundation that explains why all "data hoarders" do what's done.
Surely those who are long-time users of the Internet Archive must have known about the parameter "noindex" in items which makes it hidden from the internal search engine, although remains on their servers. This is essential cases where which the uploader had uploaded content that is controversial in certain jurisdictions but doesn't violate US law in any way and they fear that making it publicly in search engines will invite unnecessary hassles and liabilities to the Internet Archive. Modi BBC documentary might be one tenuous example.
Besides, some people might be fine that their data will be studied by archivists in the far future, say a hundred years or more, however paradoxically in terms of near future and present they are not really okay for privacy/security reasons and prefer to see it noindexed, as opposed to the dichotomy that either an item is fully public, or doesn't exist at all, which in one way or more defeats the purpose of an "archive".
Here's one of the plausible example. A political activist or so want their memories to be archived for posterity becuase they think it will be helpful for far future archivists/researchers who study their career et al. However they prefer to see it hidden but archived because right now bad actors could use that info to doxx or harass them.
There was even an informal policy by Internet Archive to noindex some YouTube videos out of copyright-related reasons, per this.
However, they had remove the function from casual usage about May or June, perhaps after the mass-scraping DDOS incident, and unhide most of the items that are used to be noindexed. Therefore I had emailed the Internet Archive to ask for an explanation and to get them reverse the decision which I think is hare-brained.
After a few going forth and backs, they finally come with this reply which I feel is totally ludicrous:
There is no bug or mistake in removing no-index settings for many Internet Archive items in the Community collection.
At no point was the Archive contacted to arrange a situation of no-indexing (or Darking) items with an intention of later release; the no-index setting was not documented for this use, and represented a security hole that was closed. Tens of thousands of items were found, being used for encrypted files hidden from the search engine, and represented a major problem, so many items have been removed or set noindex quickly.
A number of people have contacted us explaining situations where items might need to be made no-indexed, in a collection for later or timed release for example, but they've done it with communication and discussing their needs, not just uploading files under disposable accounts and then assuming the archive would keep them un-accessible in perpetuity. In some cases their requests have gotten arrangements so that community items that were noindex are noindex again, in separate collections.
A situation can theoretically exist where the original uploader can e-mail us from their e-mail address and discuss arrangements, but you've indicated you intentionally obfuscated your location and have disposed your addresses. If you're able to gain access again, you can mail through those addresses.
An additional situation is you can e-mail [email protected] if you want to report items at the archive (by identifier) that you believe might need to be removed from the archive; we receive a number of these requests throughout the months and respond according to policy.
It is as if enshittification which had plagued Reddit just not long ago, has now made its landfall to the Internet Archive. The removal of "noindex" had in my two cents, destroyed the chance to attain delicate balance between preservation and privacy, hence meaning increased vulnerability to privacy and copyright related risks.
Perhaps, if the enshittification proves irreversible, there is Texas-based permanent.org which may one day become a successor of the Internet Archive.
Here's a pseudocode for them in case they come to their senses:
noindex items if:
(
items-noindexed-by-user-in-the-past = true;
OR items-noindexed-by-IA-in-the-past = true);
AND (
items-get-reindexed-voluntarily-by-USER-before-May-2023 = false;
OR
items-get-reindexed-voluntarily-by-IA-before-May-2023 = false;
)
Bonus:
Bit bloody arrogant isn't it, assuming you'll be of historical interest in a hundred years time and that IA ought to start keeping records of you now for that.
Bonus:
Some here said that hidden files are bad because they can be abused, but that doesn't mean that legitimate uses of the function can be handwaved away with a wand, just like how a coin has two sides. Thus it appears that there's a catch-22 situation here.
However, there's always the possibility of making a census in a separate area (whether internal or not, as a json file or something) of these hidden files if the Internet Archive wants them to remain accessible in the very long run. Furthermore it's a time sink for the IA employees if all the noindex requests are set to go through them manually every time; I bet they don't appreciate having to review these and that hence distracting them from more important pursuits such as scanning books.
send them an email and loop them in
In fact, they had been briefed, both implicitly and explicitly on the relevant use case scenarios through email recently and previously. They at one point stated that Internet Archive is a library, even though there are differences between the terms "library" and "archive".
Put it simply, think about the Library of Congress and the NARA. Public access to the latter's collections is normally harder than the former. IA had already chose to die on the "archive" hill when they used that term to brand themselves, and it's not a stretch to describe it as enshittification making a landfall when they tried to deviate from that core purpose as embodied by the term.
Finally as a disclaimer, under the penalty of perjury I am in no way had condoned and supported CSA behaviors, let alone upload those heinous items to the IA.
Bonus:
You're doing something nefarious. Otherwise you'd be more forthcoming about what you're uploading. The only way you're going to be able to recruit people to your cause is if you're more transparent.
Bonus:
Please keep the snarky BS elsewhere. All I've mainly uploaded to the IA are personal memories and creations, dating as far back as childhood. With the present context. one additional benefits of noindex function is that greedy companies can't easily scrape it and profit it on the back.
More than that, I admit that I engaged in some political activism for Ukraine and Hong Kong in the past and present, the latter which had placed me under the radar of their notorious National Security Law.
You're being entitled. Just because they used the word 'archive' it doesn't mean they're obliged to follow your interpretation of what that means.
If you want data preserved, pay for it. Don't expect IA to do it for you.
It's a non-profit, period, which are sustained through donations of any kind. After all it walks and behaves as if it's a poor man's archive all along. So much for "Universal Access to All Knowledge" if you force everyone to pay in order to upload their files. Without question you all will turn sour if that logic is applied to the Wayback Machine.
Edit:
IA is not your personal cloud backup. They're not obliged to host it for you, privately or publicly. Being a political activist is neither here nor there.
Hmm. I must say that the conception as quoted above, is no longer unfamiliar because those were normally used to stonewall criticisms against unpopular changes by tech companies or platforms, most recently by spez and his supporters during the API protests. That being said, you should be careful next time when you're entertaining it at all.
By the way, Reddit had now made photos in posts pretty much unarchivable.
Bonus:
It's strange to see that long-term thinking has been unexpectedly reviled here. I suppose that many veteran archivists who would've fully understood such idea, have joined others to leave Reddit in the wake of unpopular API changes.
To understand what "long-term thinking" is, think of this: Let's say someone is a small-time actor. They might make a breakthrough one day in Hollywood, or others so much that they become a household name. Their supporters, detractors and others will certainly see overarching interests into seeing their works and deeds preserved, for scrutiny, commemoration, and any imaginable purposes.
The hypothetical actor is a believer in archiving and digital preservation before famous, so they put a lot of items in the IA for years. While they are certainly okay with all their items being discovered and publicized by digital archaeologists a hundred years later or more, they would rather prefer that some of their items being noindexed, undisturbed, perhaps for a hundred years or indefinitely, like the artifacts in the pyramids, since premature exposure would mean TMZ and other unwelcome harassments.
As can be seen in this video, the IA founder is in acceptance with the general idea of long-term thinking and preservation. It's no longer "arrogant" and "unfeasible" to expect that personal legacies be preserved at any level and degree according to the user's wishes. For now I'll end this long comment with this Twitter thread by the Long Now Foundation that explains why all "data hoarders" do what's done.