A lot of E-commerce software have data feeds that are ingested by parties like pricespy. These feeds are typically either public or made available after mutual agreements. Scraping a large amount of sites for data is just too much labor in the long run.
Kudos to Spotify for supporting libspotify and its use-cases for so many years. I'm sure it hasn't been easy and that copyright owners haven't been very fond of it.
Anecdata: I’ve found USB powered external drives to be unreliable on the Pi, even when powering the Pi with an official power adapter. I’d recommend using an external drive that comes with its own power adapter.