HackerTrans
TopNewTrendsCommentsPastAskShowJobs

mcstempel

no profile record

Submissions

How to block AI web crawlers

stytch.com
7 points·by mcstempel·letztes Jahr·0 comments

Mcp-scan: NPM-audit-style security scanner for MCPs

stytch.com
2 points·by mcstempel·letztes Jahr·0 comments

comments

mcstempel
·letztes Jahr·discuss
There are options beyond auth walls for detecting/enforcing behavior as well since these scrapers have very recognizable device signatures: https://stytch.com/blog/detecting-ai-agent-use-abuse/
mcstempel
·letztes Jahr·discuss
thanks for sharing!
mcstempel
·letztes Jahr·discuss
wow, this is wonderfully made
mcstempel
·letztes Jahr·discuss
Ah, this is great feedback -- I don't think we do enough to articulate how much we're doing beyond that simplified explanation of device fingerprinting on those docs. I'll get that page updated, but 2 main things worth mentioning:

1. We have a few proprietary fingerprint methods that we don't publicly list (but do share with our customers under NDA), which feed into our ML-based browser detection that assesses those fingerprint data points against the entire historical archives of every browser version that has been released, which allows us to discern subtle deception indicators. Even sophisticated attackers find it difficult to figure out what we're fingerprinting on here, which is one reason we don't publicly document it.

2. For a manual attacker running attacks within a legitimate browser, our Intelligent Rate Limiting (IntRL) tracks and rate-limits at the device level, making it effective against attackers using a real browser on their own machine. Unlike traditional rate limiting that relies on brute traits like IP, IntRL uses the combo of browser, hardware, and network fingerprints to detect repeat offenders—even if they clear cookies or switch networks. This ensures that even human-operated, low-frequency attacks get flagged over time, without blocking legitimate users on shared networks.
mcstempel
·letztes Jahr·discuss
CAPTCHAs have been ineffective as a true "bot detection" technique for a while as tools like anti-captcha.com allow for outsourcing it to real humans. BUT they have been successful at the economic side of raising the cost of programmatic traffic on your site (which is good enough for some use cases)

As the author of this agent detection post, we agree that CAPTCHA and vanilla browser/device fingerprinting is quickly not going to be very valuable in isolation, but we still see a lot of value in advanced network/device/browser fingerprinting

The main reason is that the underlying corpus & specificity of browser/device/network data points you get from fingerprinting makes it much easier to build more robust systems on top of it than a binary CAPTCHA challenge. For us, we've found it very useful to still have all of the foundational fingerprinting data as a primitive because it let us build a comprehensive historical database of genuine browser signatures to train our ML models to detect subtle emulations, which can reliably distinguish between authentic browsers and agent-driven imitations

That works really well for the OpenAI/BrowserBase models. Where that gets tricky is the computer-use agents where it's actually putting its hands on your keyboard and driving your real browser. Still though, it's valuable to have the underlying fingerprinting data points because you can still create intelligent rate limits on particular device characteristics and increase the cost of an attack by forcing the actor to buy additional hardware to run it
mcstempel
·letztes Jahr·discuss
You read our mind! https://stytch.com/blog/the-age-of-agent-experience/

Very much agreed that's the long-term goal, but I think we'll live in a world where most apps don't support oauth for a while longer (though I'd love for all of them to -- we're actually announcing something next week that makes this easy for any app to do)

But we're also envisioning an interim period where users are delegating to unsanctioned external agents (e.g. OpenAI Operator, Anthropic Computer Use API, etc.) prior to apps catching up and offering proper oauth
mcstempel
·letztes Jahr·discuss
LinkedIn always hits me with those frustrating custom CAPTCHAs where you have to rotate the shape 65 degrees -- they've taken a pretty blunt, high-friction approach to bot detection

I think most apps should primarily start with just monitoring for agentic traffic so they can start to better understand the emergent behaviors they're performing (it might tell folks where they actually need real APIs for example), and then go from there
mcstempel
·letztes Jahr·discuss
Hey there, I'm the author of the post. I'm actually pretty sympathetic to your viewpoint, and I wanted to clarify my stance.

I actually spent years working at a "good bot" company (Plaid), which focused on making users' financial data portable. The main reason Plaid existed was that banks made it hard for users to permission their data to other apps -- typically not solely out of security concerns, but to also actively limit competition. So, I know how the "bot detection" argument can be weaponized in unideal ways.

That said, I think it’s reasonable for app developers to decide how their services are consumed (there are real cost drivers many have to think about) -- which includes the ability to have monitoring & guardrails in place for riskier traffic. If an app couldn't detect good bots, that app also can't do things like 1) support necessary revocation mechanisms for end users if they want to clawback agent permissions or 2) require human-in-the-loop authorization for sensitive actions. Main thing I care about is that AI agent use remains safe and aligned with user intent. For your example of an anonymous read-only site (e.g. blog), I'm less worried about that than an AI agent with read-write access on behalf of a real human's account.

My idealistic long-term view though is that supporting AI agent use cases will eventually become table stakes. Users will gravitate toward services that let them automate tedious tasks and integrate AI assistants into their workflows. Companies that resist this trend may find themselves at a competitive disadvantage. Ultimately, this has started to happen with banking & OAuth, though pretty slowly.
mcstempel
·vor 2 Jahren·discuss
We built Stytch's B2B SaaS solution with this specific shortcoming in mind -- most other solutions aren't actually built with an organization-first data model (they're user-first like Auth0 but support the general concept of orgs), which makes it difficult to offer those per organization controls in an ergonomic manner.

There's some more info on our multi-tenancy data model here (https://stytch.com/docs/b2b/guides/multi-tenancy), and here's the PUT request you'd use to manage any of those org configurations: https://stytch.com/docs/b2b/api/update-organization
mcstempel
·vor 3 Jahren·discuss
You can now set up passkeys on your personal gmail, which I've found to be particularly nice for times when I'm trying to log in via webview
mcstempel
·vor 3 Jahren·discuss
Yeah, +1. Even vanilla puppeteer is pretty successful against Cloudflare
mcstempel
·vor 3 Jahren·discuss
To your point, the market will decide, but I'm hopeful passkeys will ultimately be one of the key solutions here. Already seeing a lot more app adoption (e.g. Shopify, Google, Docusign) than original webauthn given some of the UX problems that brought with it
mcstempel
·vor 3 Jahren·discuss
Co-founder of Stytch (http://stytch.com/) here -- would love to see if you think we're a fit. We have a generous free tier and we're more reasonably priced that tools like Auth0, but not as cheap as tools like Cognito + Firebase auth. If pricing is a concern, would love your feedback on your optimal cost-structure as we have some startup programs and are always looking to make pricing more compelling for early-stage projects.