A similar approach I use is google cloud sdk + ssh socks5 proxy.
Create some preemptible instances on google cloud first, then connect them with commands like "gcloud compute ssh instances-name -- -D localhost:port".
And the last step is to connect scrapper to those proxy ports over localhost.
the data is just in the folder u select when u choose local hard drive as backup location.
and the format is quite friendly for programmatically accessing.
Create some preemptible instances on google cloud first, then connect them with commands like "gcloud compute ssh instances-name -- -D localhost:port".
And the last step is to connect scrapper to those proxy ports over localhost.