Building Webhooks into Your Application: Guidelines and Best Practices (2020)(workos.com)
workos.com
Building Webhooks into Your Application: Guidelines and Best Practices (2020)
https://workos.com/blog/building-webhooks-into-your-application-guidelines-and-best-practices?src=hn
27 comments
What people totally omit when thinking about webhooks is the parallels to web sockets and client side updates. In the last round of our system updates, we have "attached" the web sockets and webhooks to the same message bus which has simplified the overall design of the system.
Yeah, they share a lot of the same challenges as they are both unlike the rest of the web stack (which is just request -> response). There are a lot of similarities, but a lot of different challenges too.
Hehe, just did a show HN[1] for Diahook (webhooks as a service) earlier today. I wonder if that's what triggered this posting. :P
Anyhow, I wish more people had better webhooks, it's such an important part of any API, so please read this blog post, understand the challenges, and provide a great webhooks experience!
[1] https://news.ycombinator.com/item?id=26399672
Anyhow, I wish more people had better webhooks, it's such an important part of any API, so please read this blog post, understand the challenges, and provide a great webhooks experience!
[1] https://news.ycombinator.com/item?id=26399672
Wow Diahook looks great!
My company is in the process of updating our webhooks, so we will be sure to check it out.
My company is in the process of updating our webhooks, so we will be sure to check it out.
Thanks! Feel free to reach out tom @ the domain. :)
Kind of wild that there's no mention of SSRF. A quick search shows it's a pretty frequent security issue in Webhooks: https://www.google.com/search?q=ssrf+webhook
This would be a good addition: make sure that your HTTP library POSTing the webhook won't follow redirects and is blocked from accessing internal domains.
This was a great read – thanks for sharing!
I've always been a huge fan of how Stripe handles their webhooks. They are pretty much doing everything listed in your article.
I've always been a huge fan of how Stripe handles their webhooks. They are pretty much doing everything listed in your article.
SSRF is a tremendous problem with webhooks. Stripe also solves this: https://fly.io/blog/practical-smokescreen-sanitizing-your-ou...
Agreed, I should add something on this. (Haven't updated the article since posting it many months ago.)
This is a pretty good article about preventing SSRF including DNS rebinding-based attacks in Go https://www.agwa.name/blog/post/preventing_server_side_reque...
Yeah, it's one of the thing that people miss when implementing webhooks naively, and I bet it's one of the things people miss when consuming them!
Hey HN - I'm the author of this post. (Founder of WorkOS)
Lots of people have written about how to scale event-based systems or how to design distributed infrastructure for queue management. But what's often overlooked is the subtle-yet-important product decisions that go into designing webhooks for a great developer experience.
We don't actually offer webhooks-as-a-service at WorkOS, but we provide webhooks to developers and I've built this feature twice now, previously at Nylas and now at WorkOS. Hopefully this article helps the next person who builds them. :)
For anyone curious how we do it at WorkOS, here's our webhooks API reference: https://workos.com/docs/reference/webhooks/connection
Lots of people have written about how to scale event-based systems or how to design distributed infrastructure for queue management. But what's often overlooked is the subtle-yet-important product decisions that go into designing webhooks for a great developer experience.
We don't actually offer webhooks-as-a-service at WorkOS, but we provide webhooks to developers and I've built this feature twice now, previously at Nylas and now at WorkOS. Hopefully this article helps the next person who builds them. :)
For anyone curious how we do it at WorkOS, here's our webhooks API reference: https://workos.com/docs/reference/webhooks/connection
I can never understand why enforcing (sane and secure) TLS and adding a (shared) secret request header isn't the usual simple answer.
As far as I know, TLS prevents all these security issues, except authentication of the webhook caller, hence the shared secret which your webhooks should check.
I've verified in the TLS specs that every issue I've seen raised by people is actually covered, yet people still think (and write) that TLS doesn't sign traffic (it does), or TLS doesn't prevent replay attacks (it does), etc.
Usually I give them pointers to the specs, and they say "it doesn't hurt to add even more security" but they're really rolling out their own encryption, which you shouldn't do.
I'd say enforce TLS and authenticate the webhook caller...
As far as I know, TLS prevents all these security issues, except authentication of the webhook caller, hence the shared secret which your webhooks should check.
I've verified in the TLS specs that every issue I've seen raised by people is actually covered, yet people still think (and write) that TLS doesn't sign traffic (it does), or TLS doesn't prevent replay attacks (it does), etc.
Usually I give them pointers to the specs, and they say "it doesn't hurt to add even more security" but they're really rolling out their own encryption, which you shouldn't do.
I'd say enforce TLS and authenticate the webhook caller...
TLS with certificate pinning could probably even take care of the authentication issue.
I'm the developer of Diahook (we do webhooks as a service), and I agree. It's just so annoying that we have to implement signing ourselves instead of just using client TLS certificates. Though the problem is that there are so many places in the stack that interact with your request before it hits user code (load balancer, web server, etc), that getting anything like this to work is wishful thinking...
Not just logging the outbound webhooks to STDOUT and hope they get ingested into logstash or something, but actually log them in your SoR/RDBMS so that you can present that data to a debugging developer.
And make sure to impose a max size. Getting a 1GB gzip bomb persisted to your DB is never fun.
Maybe a stupid question but... why not using an MQTT server instead of webhooks? Clients could be notified in RT, TLS is already backed in the protocol with client auth... the client doesn't need to expose an endpoint, only to connect to a server...
Something else Stripe and most other "serious" webhook senders do is only send Webhooks from a predefined set of IP addresses [1].
This isn't the security guarantee that signing, etc. are but it's a substantial help in adding some Security In Depth.
1 - https://stripe.com/docs/ips
This isn't the security guarantee that signing, etc. are but it's a substantial help in adding some Security In Depth.
1 - https://stripe.com/docs/ips
democratize access to webhooks -- love dat energy
This was an interesting read! If you ever decide to update this article & include tools for working with webhooks, Hookdeck(https://hookdeck.io) should be at the top of the list. Hookdeck is a complete webhook infrastructure that makes working with webhooks painless. we pride ourselves in reliability. We provide tooling for easy monitoring and troubleshooting of webhooks
AWS EventBridge announced API destinations recently [1]. I can't find anywhere in their docs about SSRF, but given it's a managed service running outside of your network it sounds like a good option. Anyone care to comment if this would be a good solution for webhooks?
1 - https://aws.amazon.com/blogs/compute/using-api-destinations-...
1 - https://aws.amazon.com/blogs/compute/using-api-destinations-...
At FastComments we send an your API token in the webhook request, and if your API doesn't return a 401 with an invalid token the integration won't activate.
I can see why you might do that, but it seems mildly hostile to me and would have broken things for me on at least one or two occasions. I know I’ve set up very simple hook arrangements before where I don’t care about authentication, at least in part because the URL is secret and it’s probably just about dumping the hook call contents for now anyway.
But at the same time, perhaps doing this would solve more problems than it creates.
But at the same time, perhaps doing this would solve more problems than it creates.
It's not hostile. It's ensuring you properly authenticate the request, and that you own the target endpoint. :)