Amazon EventBridge: The biggest thing since AWS Lambda(trek10.com)
trek10.com
Amazon EventBridge: The biggest thing since AWS Lambda
https://www.trek10.com/blog/amazon-eventbridge/
58 comments
Except you had to use CORBA or JMS to invoke them, which was pretty frustrating and expensive computationally.
CORBA and the object mapping model (similar to the issues of ORMs) is what led to the dreaded ClassCastException problems that required every service to re-deploy at once which was completely broken and indicative of the architects being engineers that hardly ever touched day to day production. The transition off of syncing object representation in a monolith and God Service Schema / Registry was necessary but by that point we had moved on to SOAP and screwed that part up once again by needing to re-deploy JARs yet again constantly and out of sync and also being unable to separately deploy and route components off which led to either more ditch digging via OSGI or by going to REST APIs... or if you didn’t get here you used earlier incarnations of object and message serializations with protocol buffers. Only with these painful lessons have we as an industry learned how to actually decouple our services’ business boundaries well enough to call them decoupled, scalable services.
Did we learn the lesson? Protobufs are getting traction, and some rising star companies started using them for data related purposes. The problem they are facing - change in the protobuf is not automatically reflected in you data schema (not like with avro), you have to recompile and redeploy pieces of your stack + orchestrate deployment so data is not lost in processing. It can be solved easily with avro, but “we want to use protos, Bla Bla Bla”
Protobufs are relatively good with backwards compatibility, no?
Thrift, ProtoBufs, and Avro solve the messaging compatibility layer problem at least (optional fields in a schema are all you need 99%+ of the time) and so far nothing out there has managed to solve any data schema incompatibility problem automatically because that’s pretty much intractable, but providing tools that make finding problems earlier certainly helps with the reality of SOA in its various incarnations.
Was it? Do you think REST/gRPC will be exciting in the 90's?
I think it was timing. Hardware and network were super slow back then.
I think it was timing. Hardware and network were super slow back then.
Agreed completely, programming patterns are just like anything else, timing is everything. I thought that ESB pattern was cool in the early 00s, today that pattern has touched off and evolved a robust ecosystem. It’s totally fair to rebrand it as a descendent of ESB, it’s fair to point out the lineage, but ecosystem evolution + pattern refinement is a distinct thing to get excited about.
Or 25 years ago building things on Tibco Rendezvous...
After reading it I can't understand how EventBridge is different to just using SQS or any old messaging service? It just sounds like a message system layered on a messaging system. Can anyone elaborate why this is significant?
It has built-in integrations so you don't have to standup a service or API Gateway to receive webhooks from their list of partners. Their initial list of partners is [1]:
- Datadog
- OneLogin
- Pagerduty
- Saviynt
- Segment
- SignalFx
- SugarCRM
- Symantec
- Whispir
- Zendesk
and of course that'll grow over time - anyone that produces events can become a partner.
[1]: https://aws.amazon.com/eventbridge/partners/
- Datadog
- OneLogin
- Pagerduty
- Saviynt
- Segment
- SignalFx
- SugarCRM
- Symantec
- Whispir
- Zendesk
and of course that'll grow over time - anyone that produces events can become a partner.
[1]: https://aws.amazon.com/eventbridge/partners/
So kind of an IFTT or Zapier piped directly into your AWS backend?
Yeah, I also thought of IFTT when I saw this:
https://ifttt.com/services
Reminds me a bit of Apache Camel then - let's you easily route end points from all kinds of different protocols as if they are messages.
Maybe I never looked at it hard enough, but I couls never figure out what Camel was for.
It's one of those things that you don't know you need ... until you get half way to re-inventing it yourself and then you realise you needed it. Wrt this, probably the more relevant part is all the end point integrations:
https://camel.apache.org/components.html
So it makes it super easy to, say, connect an inbound email to an ftp server so that all the messages get saved as files.
https://camel.apache.org/components.html
So it makes it super easy to, say, connect an inbound email to an ftp server so that all the messages get saved as files.
Camel was libraries that codified the the Enterprise Integration Patterns (https://www.enterpriseintegrationpatterns.com/ - channels, messages, service activators, etc...) that allowed you to code against the abstractions rather than the underlying implementations (JMS w/ topics & queues, AMQP w/ exchanges, etc..)
Spring offered a competing set of libs with Spring Integration.
Those we mostly just libraries, though, so you have something running under them for distribution (like a message broker or service bus or, these days, a cloud managed service).
Spring offered a competing set of libs with Spring Integration.
Those we mostly just libraries, though, so you have something running under them for distribution (like a message broker or service bus or, these days, a cloud managed service).
> so you have something running under them for distribution (like a message broker or service bus or, these days, a cloud managed service).
Actually Apache Camel doesn't really care if you have a message broker or not. You can set it up without one and still use half its functionality. (eg: connect a file arriving in a dropbox account to an outbound SMTP etc).
Actually Apache Camel doesn't really care if you have a message broker or not. You can set it up without one and still use half its functionality. (eg: connect a file arriving in a dropbox account to an outbound SMTP etc).
Eek! Missing Salesforce!!!
This looks similar to Azure Event Grid, which I'm more familiar with. If that's true, the value prop is that one event handler can be webhook based while another handler can be queue based, and you can add and remove these different kinds of handlers without making any producer-side changes. It's quite nice
So SNS?
And SQS rolled into one with lots of click and play integrations
Not a ton of outside integrations, but this library creates and maintains an esb using sqs and sns with just a few annotations in java. Uses json as the message and maps to pojo in the application.
https://bitbucket.org/pythagorasio/java-message-bus/src/mast...
Its also published on maven central. Io.pythagoras.messagebus
Disclosure: I’m the primary author.
https://bitbucket.org/pythagorasio/java-message-bus/src/mast...
Its also published on maven central. Io.pythagoras.messagebus
Disclosure: I’m the primary author.
How is this "SNS and SQS rolled into one"? What's the SQS-like part in this? This seems very similar to SNS, with some additional flexibility around event filtering, and the added bonus of various SaaS integrations that can replace traditional webhooks.
EventBridge is a PubSub system with different tradeoffs and operational constraints and behaviors.
The cool new features are: (1) It allows serverless dynamic routing based on message content which auto-scales to match usage. (2) It supports SaaS partner events. (3) It supports publishing events based on a scheduled configuration, which can be used to trigger cron jobs or other system processes.
So it is just a messaging system, but one that might be more appropriate to certain use cases over using SQS or SNS or Kinesis, etc.
The article thinks that the ability to have SaaS partner publish events will be a huge deal and is a killer feature of EventBridge.
The cool new features are: (1) It allows serverless dynamic routing based on message content which auto-scales to match usage. (2) It supports SaaS partner events. (3) It supports publishing events based on a scheduled configuration, which can be used to trigger cron jobs or other system processes.
So it is just a messaging system, but one that might be more appropriate to certain use cases over using SQS or SNS or Kinesis, etc.
The article thinks that the ability to have SaaS partner publish events will be a huge deal and is a killer feature of EventBridge.
Under the hood it's probably exactly that.
Pretty sure AWS is late to the party and just now achieving feature parity with Azure. Azure Event Grid came out at the beginning of 2018.
However, I suppose from an AWS cloud standpoint this great news :)
I would imagine this replaces a lot of SNS and SQS combo architectures in AWS?
However, I suppose from an AWS cloud standpoint this great news :)
I would imagine this replaces a lot of SNS and SQS combo architectures in AWS?
it looks like it only replaces SNS no? As some of the examples are Lambdas and SQS queues that you can publish to. We have a small Go service that basically maps endpoints to SNS topics which then publishes events to SQS or Lambda or some other internal endpoint.
Don't see how this is much different other than provided integrations with popular services out of the box.
Is it also like Azure Logic App?
Everything old is new again!
https://en.wikipedia.org/wiki/Yahoo!_Pipes
But in all seriousness we've got half a dozen use cases for this just off the top of my head.
https://en.wikipedia.org/wiki/Yahoo!_Pipes
But in all seriousness we've got half a dozen use cases for this just off the top of my head.
I loved yahoo pipes. Wonder why it never took off. You could do SQL style query on dozens of different services.
How do you test this in dev environment?
By subscribing a dev AWS account to the EventBridge.
Live subscriptions sound like like staging, not dev
Hoping for localstack (https://github.com/localstack/localstack) support
Just saw Amazon CTO Dr Werner Vogels present this at their Summit here in NYC. He was very excited about it!
Their other big reveal was for new AWS CDK services allowing programmable config at scale in a few lines of code
https://aws.amazon.com/blogs/aws/aws-cloud-development-kit-c...
Their other big reveal was for new AWS CDK services allowing programmable config at scale in a few lines of code
https://aws.amazon.com/blogs/aws/aws-cloud-development-kit-c...
Amazon is a logistics company, not a cloud company. Their product strategy is spray and pray. "You want a DB? We have 10!" GC has fewer, more stable, high quality offerings. Amazon is like their store "Want shoes? We got millions." Google is like their site "You want search, we do search."
As people say in sports when someone is trash talking when they are losing: “Scoreboard”
The AWS approach (and Azure) is clearly working. And the reality is that people want multiple approaches, especially enterprises, which are typically migrating applications to the cloud and can’t just rewrite everything for Google’s random proprietary platform.
Also, I’ll note that GCP is pretty aggressive about EOLing platforms. Good luck with that in the enterprise market.
The AWS approach (and Azure) is clearly working. And the reality is that people want multiple approaches, especially enterprises, which are typically migrating applications to the cloud and can’t just rewrite everything for Google’s random proprietary platform.
Also, I’ll note that GCP is pretty aggressive about EOLing platforms. Good luck with that in the enterprise market.
You seem to have confused googles consumer platform policy and Googles enterprise policy. They are very different.
That’s like saying AWS is low quality because you bought a fake widget off Amazon.com?
That’s like saying AWS is low quality because you bought a fake widget off Amazon.com?
Google Cloud Messaging: https://developers.google.com/cloud-messaging
App Engine: https://cloud.google.com/appengine/docs/deprecations/
...
App Engine: https://cloud.google.com/appengine/docs/deprecations/
...
Several of those are showing multiple year deprecation policies. I think the min from notice to shutdown is like 2 years?
If you really want to build something and never touch it again, you need to go in prem. Good luck with being 5 years behind in security updates.
If you really want to build something and never touch it again, you need to go in prem. Good luck with being 5 years behind in security updates.
We have a very small Golang service that exposes some endpoints and basically just forwards incoming events to SNS which then publishes to subscribers like SQS/Lambda/whatever. This small service basically never fails as it's only dependency is SNS. I don't really see how this EventBridge is much more than that?
I have a very similar setup what you have. I think what event bridge offers, is the ability to integrate with anything which can reproduces event. They are trying to expand their base beyond AWS ecosystem. Datadog, Zendesk, OneLogin are great services which could be helpful, to begin with. Still time will tell how popular it gets.
Big win for serverless, especially in price:
"There is no charge for events published by AWS services"
"There is no charge for events published by AWS services"
Is it the same as Azure EventGrid?
For those confused, I suggest you go read the official doc: https://aws.amazon.com/eventbridge/faqs/ . Let me quote the important bits:
> EventBridge was formerly called Amazon CloudWatch Events. It includes new features that enable you to receive events from SaaS partners and your own applications. Existing CloudWatch Events users can access their existing default bus, rules, and events in the new EventBridge console and in the CloudWatch Events console. EventBridge uses the same CloudWatch Events API, so all of your existing CloudWatch Events API usage remains the same.
So EventBridge is CloudWatch Events. The difference is that now you can publish your own events and specify your own routing rules for them. As well as supporting 3rd party events from certain SaaS partners.
It is limited to 400 events per second, 5 targets per rule, and 100 rule per account. Consuming events is throttled at 750 per second. Events take about half a second to reach targets. Some of these limits can be increased by asking for higher limits. AWS published events, so existing CloudWatch Events don't count towards these and are unlimited.
What about SNS?
> Amazon SNS is recommended when you want to build an application that reacts to high throughput or low latency messages published by other applications or microservices (as Amazon SNS provides nearly unlimited throughput), or for applications that need very high fan-out (thousands or millions of endpoints)
> At launch, Amazon EventBridge is has limited throughput (see Service Limits) which can be increased upon request
> Amazon EventBridge is the only event-based service that integrates directly with third-party SaaS partners
> Amazon EventBridge uses a defined JSON-based structure for events, and allows you to create rules that are applied across the entire event body to select events to forward to a target
Also, the targets are not exactly same yet. For example, only SNS can publish to Mobile SMS, Email, or HTTP/S endpoints for now.
And SNS allows any type of payloads, while EventBridge forces you into JSON.
Finally, the reason this specifies serverless is because of the dynamic content based routing using configurable rules. This means you don't need to manage your own routing. So say you want to pair it with Lambda, you don't waste Lambdas starting then realizing the event isn't relevant for them, you can just setup rules for that. So it provides a serverless routing mechanism that auto-scales and is decoupled from producer and consumer. That's pretty cool!
Hope that helps!
> EventBridge was formerly called Amazon CloudWatch Events. It includes new features that enable you to receive events from SaaS partners and your own applications. Existing CloudWatch Events users can access their existing default bus, rules, and events in the new EventBridge console and in the CloudWatch Events console. EventBridge uses the same CloudWatch Events API, so all of your existing CloudWatch Events API usage remains the same.
So EventBridge is CloudWatch Events. The difference is that now you can publish your own events and specify your own routing rules for them. As well as supporting 3rd party events from certain SaaS partners.
It is limited to 400 events per second, 5 targets per rule, and 100 rule per account. Consuming events is throttled at 750 per second. Events take about half a second to reach targets. Some of these limits can be increased by asking for higher limits. AWS published events, so existing CloudWatch Events don't count towards these and are unlimited.
What about SNS?
> Amazon SNS is recommended when you want to build an application that reacts to high throughput or low latency messages published by other applications or microservices (as Amazon SNS provides nearly unlimited throughput), or for applications that need very high fan-out (thousands or millions of endpoints)
> At launch, Amazon EventBridge is has limited throughput (see Service Limits) which can be increased upon request
> Amazon EventBridge is the only event-based service that integrates directly with third-party SaaS partners
> Amazon EventBridge uses a defined JSON-based structure for events, and allows you to create rules that are applied across the entire event body to select events to forward to a target
Also, the targets are not exactly same yet. For example, only SNS can publish to Mobile SMS, Email, or HTTP/S endpoints for now.
And SNS allows any type of payloads, while EventBridge forces you into JSON.
Finally, the reason this specifies serverless is because of the dynamic content based routing using configurable rules. This means you don't need to manage your own routing. So say you want to pair it with Lambda, you don't waste Lambdas starting then realizing the event isn't relevant for them, you can just setup rules for that. So it provides a serverless routing mechanism that auto-scales and is decoupled from producer and consumer. That's pretty cool!
Hope that helps!
[deleted]
I would really like to see how this compares to Firebase. It sounds really similar in that you subscribe to events and then AWS or Google handles the difficult tasks of routing to the right places.
“It re-establishes how the greater ecosystem of software integrates with your business logic, in addition to defining an excellent standard for your own services to leverage.”
But what is it?
But what is it?
It's PubSub with dynamic message routing, but where each AWS account will also now be able to make themselves a consumer of events published by 3rd party SaaS. Say you want to start a Lambda whenever a Zendesk ticket is opened, you can now do that without having to setup any hosts or write any code.
[deleted]
Is “AppleEvents for Serverless” a reasonable analogy for this? It’s an inter-process event bus?
I think i’m getting old... was telling coworkers that their fancy micro services existed 15 years ago in the form of EJB and service bus was a thing. And look, here it goes again