tw4l · HackerTrans

3 वर्ष पहलेdiscuss

They could! They'd have to also change how faculty review and tenure work, to incentivize faculty to publish in alternative open access journals instead of the top (paywalled) journals in their field. Otherwise, faculty will continue publishing with the for-profit publishers even if there's alternatives available.

tw4l

3 वर्ष पहलेdiscuss

Former academic librarian and archivist here. It's true that university libraries are changing their shape, as you say. More study space, fewer physical collections and book purchases (and, in some areas, universities moving toward sharing their physical collections in remote offsite storage and only calling back materials when requested), and much more investment in electronic resources like journal and database access.

It's a complicated issue, but in general library budgets have been consistently shrinking for years/decades, even as the subscription costs for journal access from big for-profit like Elsevier skyrockets. It's even gotten to the point where universities with endowments larger than many nations' GDPs are struggling to afford publisher prices (see https://www.theguardian.com/science/2012/apr/24/harvard-univ...).

Decreasing library budgets plays into this, but at its heart I think that there's a crisis of purpose as well. By not investing in open access publishing infrastructure themselves and not modifying the incentives for faculty review and tenure, universities have put themselves at the mercy of these for-profit publishers and are suffering for it. It's hard to blame faculty who are vying for one of the ever-diminishing tenured spots for publishing their articles in the best journals they can, regardless of whether they'll be behind a paywall, especially when the bar for tenure keeps getting higher and higher.

So we find ourselves in a situation where universities and public funding agencies are paying academics to conduct research and do peer review, then faculty are giving the results to a for-profit publisher, who then sells access to the work right back to them at an astronomical fee.

I'm hopeful if not optimistic that at some point, universities might choose to break out of the cat-and-mouse game they're in by investing in open access publishing and incentivizing their faculty to use it rather than continuing to feed for-profit publishers' wild profits. I've seen some (too limited) action on the first part of that but pretty much none on incentivization part. Certain funding agencies are starting to require open access publication as a condition of some grants, but the change has been slow.

tw4l

3 वर्ष पहलेdiscuss

Unfortunately not so easy because archiveweb.page relies heavily on CDP. But contributions are welcome if you want to have a go at it!

tw4l

3 वर्ष पहलेdiscuss

We experimented moving to Playwright but Playwright doesn't handle long-running browser sessions well, as the devs have (maybe rightly) prioritized its use for testing over archival use cases and want you to spin up a browser each time. For archival purposes, that doesn't work as well because we're not able to save the browser profile to retain cookies such as login credentials, so we've moved back to using Puppeteer for now.

tw4l

3 वर्ष पहलेdiscuss

(Disclaimer: I work at Webrecorder)

Our automated crawler browsertrix-crawler (https://github.com/webrecorder/browsertrix-crawler) uses Puppeteer to run browsers that we archive in by loading pages, running behaviors such as auto-scroll, and then recording the request/response traffic in the WARC format (by default in Webrecorder tools, then packaged into a portable WACZ file: https://specs.webrecorder.net/wacz/1.1.1/). We have custom behaviors for some social media and video sites to make sure that content is appropriately captured. It is a bit of a cat-and-mouse game as we have to continue to update these behaviors as sites change, but for the most part it works pretty well. The crawler also has some job queuing functionality, supports multiple workers/browsers, and is highly configurable to set timeouts, page limits, etc.

The trickier part is in replaying the archived websites, as a certain amount of re-writing has to happen in order to make sure the HTML and JS are working with archived assets rather than the live web. One implementation of this is replayweb.page (https://github.com/webrecorder/replayweb.page), which does all of the rewriting client-side in the browser. This sets you interact with archived websites in WARC or WACZ format as if interacting with the original site. replayweb.page can run locally in your browser without needing to send any data to a server or can be hosted, including in an embedded mode.

(edit: fixed typos)

comments