In the meantime, as you say, we’re now going through and evaluating other vendors for each component that CF provides - which is both unfortunate, and a frustrating use of time, as CF’s services “just worked” very well for a very long time.
But it's perfect HN bait, really. The title is spicy enough that folks will comment without reading the article (more so than usual), and so it survives a bit longer before being flagged as slop.
Great post and a great little tool. Some of my experience using these techniques in production:
1. Trilateration mostly doesn't work with internet routing, unlike GPS. Other commenters have covered this in more detail. So the approach described here - to take the closest single measurement - is often the best you can do without prior data. This means you need a crazy high distribution of nodes across cities to get useful data at scale. We run our own servers and also sponsor Globalping and use RIPE Atlas for some measurements (I work for a geo data provider), yet even with thousands of available probes, we can only accurately infer latency-based location for IPs very close to those probes.
2. As such, latency/traceroute measurements are most useful for verifying existing location data. That means for the vast majority of IP space, we rely on having something to compare against.
3. Traceroute hops are good; the caveat being that you're geolocating a router. RIPE IPmap already locates most public routers with good precision.
4. Overall these techniques work quite well for infrastructure and server IP addresses but less so for eyeball networks.
Much of the internet still does not support IPv6, so most providers will give you an IPv4 address. In fact only a few providers even support IPv6 at all.
Even with IPv6 it's not a huge problem. With a few samples we can know that a provider is operating in a given /64 or /48 or even /32 space, and can assign a confidence level that the range is used for VPNs.
Working on improving the data pipeline for https://iplocate.io - an IP intelligence service I've worked on since 2017. A couple of recent focuses:
1. VPN and proxy detection. We already track dozens of providers, but we can do better here. There's also a bunch of metadata we collect as part of this process which we don't currently surface, so I'm looking at what else we can bring to our databases and free API.
2. Better detail and evidence on how we build and test our own geolocation database, which we create from scratch. There's been a recent trend of misinformation about geo accuracy, including from some other providers, so I want to better explain the accuracy (and inaccuracy) of various techniques, our policy for when we prefer certain data, and so on.
(Open to partnerships for any folks looking for a new provider!)
There are plenty of VPN and proxy detection services, either as a service (API) or downloadable database, which are surprisingly comprehensive. Disclaimer: I’ve run one since 2017. Years on, our primary data source is literally holding dozens of subscriptions to every commercial provider we can find, and enumerating the exit node IP addresses they use.
There are also other methods, like using zmap/zgrab to probe for servers that respond to VPN software handshakes, which can in theory be run against the entire IP space. (this also highlights non-commercial VPNs which are not generally the target of our detection, so we use this sparingly)
It will never cover every VPN or proxy in existence, but it gets pretty close.
Most of these providers are in fact open about the fact that these locations are “virtual”, so it’s misleading to say they don’t match where they claim to be.
There is however an interesting question about how VPNs should be considered from a geolocation perspective.
Should they record where the exit server is located, or the country claimed by the VPN (even if this is a “virtual” location)? In my view there is useful information in where the user wanted to be located in the latter case, which you lose if you only ever report the location of servers.
(disclaimer: I run a competing service. we currently provide the VPN reported locations because the majority of our customers expect it to work that way, as well as clearly flagging them as VPNs)
Working on improving the data pipeline for https://iplocate.io - an IP intelligence service I've worked on since 2017.
Recent focus has been on geolocation accuracy, and in particular being able to share more data about why we say a resource is in a certain place.
Lots of folks seem to be interested in this data, and there's very little out there. Most other industry players don't talk about their methodology, and those that do aren't overly honest about how X or Y strategy actually leads to a given prediction, or the realistic scale or inaccuracies of a given strategy, and so on. So this is an area I'm very interested in at the moment and I'm confident we can do better in. And it's overall a fascinating data challenge!
ISPs have no obligation, although the ubiquity of sites and apps relying on IP geolocation mean that ISPs are incentivized to provide correct info these days.
I run a geolocation service, and over the years we've seen more and more ISPs providing official geofeeds. The majority of medium-large ISPs in the US now provide a geofeed, for example. But there's still an ongoing problem in geofeeds being up-to-date, and users being assigned to a correct 'pool' etc.
Mobile IPs are similar but are still certainly the most difficult (relative lack of geofeeds or other accurate data across providers)
I thought this was going to be an analysis of articles that are clearly AI-generated.
I feel like that’s an increasing ratio of top posts, and they’re usually an instant skip for me. Would be interested in some data to see if that’s true.
G2, Sourceforge (yes, that one), and Gartner’s Capterra/GetApp/SoftwareAdvice all have the same business plan: charge vendors $x,xxx+ per month to outrank other vendors in their made up categories.
Of course, you can technically list for free.
But look! For the low low price of $x,xxx per month, now you can show one of 40 tailor-made award icons on your site!
Or, unlock the privilege of showing “user reviews” from our site on your site! (of course if you had managed to get reviews independently, you’re not allowed to use the widget without paying)
Don’t have reviews? Ah, I forgot to mention. The $x,xxx plan also comes with “review generation” — we’ll pay users to write reviews for you!
Oh, and on an unrelated note, the $x,xxx plan just also happens to unlock dofollow links across each of those 40 made up categories, which all rank highly in google. And the $xx,xxx plan means that - user ratings aside - you can end up at the top of those categories.
It’s hard to describe it other than the author says: a grift. Seeing those logos on other companies sites are now a huge turn off to me personally, and I haven’t yet capitulated for my own SaaS, but I suspect this isn’t the feeling of the execs they seek to target. Or maybe it is, and it’s just the price of doing business.
The readme covers connection draining with Traefik which should solve one of the issues the author mentions