sir-alien · HackerTrans

Can we not just have a whitelist for allowed crawlers and ban the rest by default? Then places like DuckDuckGo and Google can provide a list of IP addresses that their crawlers will come from. Then simply just don't include major LLM providers like OpenAI

sir-alien

昨年議論

logins are more easily banned, and highly complex captchas for signup needs a human to signup and solve. As long as it's easier to get banned than it is to signup it will at least deter.

sir-alien

昨年議論

It's going to get to the point where everything will be put behind a login to prevent LLM scrapers scanning a site. Annoying but the only option I can think of. If they use an account for scraping you just ban the account.

sir-alien

2 年前議論

Could you not instead of using one nmap process to scan 200+ addresses, just instead initiate 200+ nmap processes scanning just one IP.

Still effectively hits your spoofing system but now they bring their time back down to what it would take to scan a single IP address.

I'm sure there are many other ways around this but like all security it's merely a case of making it difficult enough that an attacker would need serious incentive to make the attack.

sir-alien

2 年前議論

> “This is an isolated, ‘one-of-a-kind occurrence’ that has never before occurred with any of Google Cloud’s clients globally,” Google Cloud CEO Thomas Kurian and UniSuper CEO Peter Chun said in a joint statement obtained by The Guardian May 8. “This should not have happened. Google Cloud has identified the events that led to this disruption and taken measures to ensure this does not happen again.”

But it's not a one of a kind thing...

Sure, one of a kind at this scale but I've heard numerous stories of GCP/AWS terminating accounts with no explanation even when asked for one. However because the customer is small, it seems like it just vanishes in the noise and nothing comes of it. It's quite simple, use a cloud provider as a backup but don't trust your primary data with any cloud provider.

4 copies, 2 with completely different cloud providers, with 2 additional copies being far away from any cloud provider each using different storage medium.

sir-alien

2 年前議論

Many are speculating that this will not be fixed by AWS (by design) however now that this has been discovered AWS will "need" to repair this flaw or they will start incurring customer flight to more secure or cheaper services.

The question is more about how long AWS are going to take to fix this issue and how many DDoS bills will they forgive.

sir-alien

2 年前議論

The Wizard 8x22B is definitely for the high end, even the 2bit version. I attempted to run it on a workstation with RTX3090 and the performance was as bad as 1 word per 2 seconds. Probably a good candidate for a Groq accelerator.

コメント