We've had an interesting experience on the interviewing side of this. Asking a system design question and getting answers involving multiple layers of caching, pub/sub, event-driven whatever/etc.. when the real answer is to just use postgres. It handles the scale we're asking about. We know it, as that's what we use internally.
I've only worked at my startup so I can't comment on scale elsewhere, but if our simple architecture can handle 30k requests per second, I think it can handle most other companies scale too.
It may look like they're all easily interchangable because the UI and actions are similar (you have a viewport and can do extrudes, etc..) but fundamentally, they're all working on very different objects at their core. Blender and 3DS Max are the most alike, but Zbrush is an entirely different paradigm and so is parametric CAD. An extrude in Blender is massively different from a pad in FreeCAD.
Maybe, with a ton of time and effort the blender UI could be abstracted from most of the box-modeling approach and then pasted over a different paradigm, but It'd take tens of thousands of hours I imagine,.
I agree that write-actions should be protected, especially now when every other person online is a bot. As for read-actions, I'll continue to profit off those being protected too but I wouldn't be too bothered if something suddenly changed and all content across the internet was a lot easier to access programmatically. I think only harm can come from that data being restricted to the huge (nefarious) companies that can pay for that data or negotiate backroom deals.
I live in the UK and can't view a large portion of the internet without having to submit my ID to _every_ site serving anything deemed "not safe the for the children". I had a question about a new piercing and couldn't get info on it from Reddit because of that. I try using a VPN and they're blocked too. Luckily, I work at a copmany selling proxies so I've got free proxies whenever I want, but I shouldn't _need_ to use them.
I find it funny that companies like Reddit, who make their money entirely from content produced by users for free (which is also often sourced from other parts of the internet without permission), are so against their site being scraped that they have to objectively ruin the site for everyone using it. See the API changes and killing off of third party apps.
Obviously, it's mostly for advertising purposes, but they love to talk about the load scraping puts on their site, even suing AI companies and SerpApi for it. If it's truly that bad, just offer a free API for the scrapers to use - or even an API that works out just slightly cheaper than using proxies...
My ideal internet would look something like that, all content free and accessible to everyone.
Our postgres replication suddenly stopped working and it took three of us hours - maybe days - of looking through the postgres source before we actually accepted it wasn't us or our hosting provider being stupid and submitted a ticket.
I can't imagine the level of laziness or entitlement required for a student (or any developer) to blame their tools so quickly without conducting a thorough investigation.
I've taken time today to do this. With some of your suggestions, I am seeing an improvement in it's ability to do some of the grunt work I mentioned. It just saved me an hour refactoring a large protocol implementation into a few files and extracted some common utilities. I can recognise and appreciate how useful that is for me and for most other devs.
At the same time, I think there's limitations to these tools and that I wont ever be able to achieve what I see others saying about 95% of code being AI written or leaving the AI to iterate for an hour. There's just too many weird little pitfalls in our work that the AI just cannot seem to avoid.
It's understandable, I've fallen victim to a few of them too, but I have the benefit of the ability to continuously learn/develop/extrapolate in a way that the LLM cannot. And with how little documentation exists for some of these things (MASQUE proxying for example) anytime the LLM encounters this code it throws a fit, and is unable to contribute meaningfully.
So thanks for your suggestions, it has made Claude better and clearly I was dragging my feet a little. At the very least, it's freed up a some more of my time to work on the complex things Claude can't do.
We have an in-house, Rust-based proxy server. Claude is unable to contribute to it meaningfully outside of grunt work like minor refactors across many files. It doesn't seem to understand proxying and how it works on both a protocol level and business logic level.
With some entirely novel work we're doing, it's actually a hindrance as it consistently tells us the approach isn't valid/won't work (it will) and then enters "absolutely right" loops when corrected.
I still believe those who rave about it are not writing anything I would consider "engineering". Or perhaps it's a skill issue and I'm using it wrong, but I haven't yet met someone I respect who tells me it's the future in the way those running AI-based companies tell me.
I don't mean that you can't do it, just that there is no company offering it so right now those are the only two options.
It's something we're experimenting with currently. the other commenter is right about apple products, but on android, desktop, etc... it's pretty easy.
Not really. You can have 100,000 IPs from proxies or use VPNs and have only 5 egress IPs.
Anybody who wants to stop the scraper could get browser fingerprints, cross reference similar ones with those IPs and quite safely ban them as its highly likely theyre not a legitimate customer.
Its a lot harder to do it for the 100k IPs because those IPs will also have legitimate customer traffic on them and its a lot more likely the browser fingerprint could just be legitimate.
The risk of false postives (blocking real people) is usually higher than just allowing the scrapers and the incetives of a lot of sites arent aligned with stopping scrapers anyway. Think eccommerce, do they _really_ care if the product is being sold to scalpers or real customers? If anything, that behaviour can raise perception of their brand, increase demand, increase prices.
This tool should have less false positives than most, so maybe it will see more adoption than others (TCP fingerprinting for example) but I dont think this is going to affect anyone doing scraping seriously/at scale.
Web scrapers maybe aren't "bad actors", but many sites dont want them. They'll use tons of TCP proxies which route them through a rotating pool of end user devices (mobiles, routers, etc...). Its not really possible to block these IPs as you'd also be blocking legitimate customers so other ways to detect and block are required.
BrighData offer H3/QUIC but only in beta and you have to contact their sales team as far as I'm aware.
We (PingProxies) might be the only company to offer H3 to the proxy/QUIC to the target using the CONNECT-UDP method publicly. Although, it is in beta/unstable until I merge my changes into Rust's H3 library.
If you wanna play around with it, email me and I'll get you some credit. I think theres potential for stealth since outdated proxy clients/servers mean automated actors never use H3.
The proxy industry is full of another 100 companies saying they offer H3/QUIC, when they mean UDP proxying using SOCKS. I suppose the knowledge gap and what customers care about (protocol to end target) is very different to what I care about (being right/protocol to the proxy server).
I mentioned this in a podcast recently; fingerprinting of proxy servers using QUIC is a lot harder as UDP doesnt have enough headers to allow for unique characteristics like a TCP does.
Theres no way to include a timestamp in a UDP datagram so all timestamps received would be from the client machine.
The most common method of proxying with residential proxies is still CONNECT tunnels and from my tests it catches a resi-proxy about 50% of the time. More with tuning of the score thresholds.
A request to a HTTPS target through a proxy will use a CONNECT request to establish a tunnel to the target.
This tunnel operates at layer 3, where the client sends TCP segments to the proxy, the server unpacks the segments and then repacks them into new segments to send to the end target. These new TCP segments will contain the timestamp of when they were created.
The HTTP request sent through those segments is unmodified, meaning it will contain the original timestamp from the client machine.
The newer timestamp on the TCP segments means there is a mismatch between the TCP RTT and the HTTP RTT.
And then some really weird stuff: "he may also have a penchant for video games, excessive drinking, and skipping work".
Guessing my exact location - Oslo Opera House - was impressive though.