- Don't use split DNS. Don't use any special internal or dev domain. Leave it to your infrastructure to route/NAT those public IPs to your internal network.
- Don't use the HTTP-01 challenge. Use DNS-01.
- Don't run your own internal CA. Use Let's Encrypt. If you care about name leakage (CT Logs), use wildcard certs. Use a central reverse proxy/load balancer for termination.
There are ways to work around this. I've just tested this: I've used the URL inspection tool of Google Search Console to fetch a URL from my website, which I've configured to redirect to a paywalled news article. Turns out the crawler follows that redirect and gives me the full source code of the redirected web site, without any paywall.
That's maybe a bit insane to automate at the scale of archive.today, but I figure they do something along the lines of this. It's a perfect imitation of Googlebot because it is literally Googlebot.
Exactly. If I was an admin of a popular news website I would try to archive some articles and look at the access logs in the backend. This cannot be too hard to figure out.
Because it works too reliably. Imagine what that would entail. Managing thousands of accounts. You would need to ensure to strip the account details form archived peages perfectly. Every time the website changes its code even slightly you are at risk of losing one of your accounts. It would constantly break and would be an absolute nightmare to maintain. I've personally never encountered such a failure on a paywalled news article. archive.today managed to give me a non-paywalled clean version every single time.
Maybe they use accounts for some special sites. But there is definetly some automated generic magic happening that manages to bypass paywalls of news outlets. Probably something Googlebot related, because those websites usually give Google their news pages without a paywall, probably for SEO reasons.
Kinda off-topic, but has anyone figured out how archive.today manages to bypass paywalls so reliably? I've seen people claiming that they have a bunch of paid accounts that they use to fetch the pages, which is, of course, ridiculous. I figured that they have found an (automated) way to imitate Googlebot really well.
I hate NAT with a passion. It's a terrible technology, whose disruptive nature has probably prevented any novelty on the transport layer. But this article is oversimplifying things.
It is well known that NAT is not meant for security and that NAT is not a firewall. But one cannot deny that it implicitly brings some "default" security to the table. With NAT it's basically impossible to screw you over because there is no meaningful practical way to allow inbound connections without the client explicitly defining them (port forwarding). With IPv6, you could have a lazy vendor that does not do any firewalling or a has a default allow policy or maybe buggy firewall. With NAT that is not possible. There is no lazy/buggy NAT implementation that allows inbound connections for your entire network, because it is technically not possible. When a NATting device receives a packet with a destination port that has not previously been opened by a client, it does not decide to drop this packet because of a decision by the vendor. It drops the packet because there is simply no other option due to the nature of NAT. That is what people mean when they talk about the inherent "security" of NAT.
Again, NAT is terrible. We need to finally get rid globally of IPv4 and all the NATting that comes with it. But let's keep it to the facts.
For me, type hints are mainly useful because they're the only reliable way to get decent IDE auto-completion. Beyond that, they feel like a bolted-on compromise that goes against the spirit of Python. If you really need strict typing, you're probably better off using a statically typed language.
The LaTeX community is astonishingly good at gatekeeping. I can't think of another field where the adoption of a clearly superior modern alternative has been so slow. For some reason, they seem to take pride in clinging to a 50-year-old typesetting system—with its bloated footprint, sluggish compilation, incomprehensible error messages, and a baroque syntax that nobody truly understands. People have simply learned just enough to make it work, and now they treat that fragile familiarity as a virtue.
Are there reported incidents where somebody was caught with such sophisticated techniques? It seems like every time I read about how the FBI caught some big darknet criminal it was pretty much always some trivial mistake on the criminal's side.
They don’t. You put the addresses in public DNS.