Ask HN: How has your experience with AWS Lambda been?
22 comments
In my experience so far (devOps and managing infrastructure) it is a great way to run one-off scripts in Python, JS and now Golang to invoke other services (eg. step functions, or other lambdas), publish to SNS topics or describe tables in dynamo, you get the gist. The advantage is that you only get charged for the time you use it, and their free tier is generous, so our bills from Lambda are negligible.
The paradigm I use is that of treating Lambda as "glue" between services, and it has worked out amazingly well for that. Quick invocations, seemingly infinite scaling. CW logs (monitoring) could be better, but tools like Apex help with parsing them.
The paradigm I use is that of treating Lambda as "glue" between services, and it has worked out amazingly well for that. Quick invocations, seemingly infinite scaling. CW logs (monitoring) could be better, but tools like Apex help with parsing them.
AWS lambda should be thought of as functions that can be scaled independently.
I can write the functions in any one of the languages that are supported and they are all interlace-able through an interface. If I need to use ML and prefer python libraries, I can do that and call python from a nodejs function or vice-versa. All this without having to worry abt keeping a server up and secure and scaling.
There are issues with loading times and cost, but easily the best thing that happened for me as a single dev proving out a product MVP.
I can write the functions in any one of the languages that are supported and they are all interlace-able through an interface. If I need to use ML and prefer python libraries, I can do that and call python from a nodejs function or vice-versa. All this without having to worry abt keeping a server up and secure and scaling.
There are issues with loading times and cost, but easily the best thing that happened for me as a single dev proving out a product MVP.
I tried using API Gateway and Lambda to create some dumb endpoints late last year (most just return static data with some returning updateable data).
All endpoints/operations exposed through API Gateway.
Thought it would be a lot simpler to set up than it was. It's pretty convoluted doing API Gateway -> Lambda, especially when you make changes.
I had thought it would be way simpler. single click define endpoint, then just write some code.
The AWS UI most definitely got in the way trying to accomplish this.
All endpoints/operations exposed through API Gateway.
Thought it would be a lot simpler to set up than it was. It's pretty convoluted doing API Gateway -> Lambda, especially when you make changes.
I had thought it would be way simpler. single click define endpoint, then just write some code.
The AWS UI most definitely got in the way trying to accomplish this.
Background: I just started a systems integration project that leverages Lambda pretty heavily as part of a messaging API. Behind an API Gateway Lambda functions will be responsible for processing initial message requests, routing the requests to certain SQS queues based on certain criteria, and then making requests to other systems as messages come into the various queues. Each of these parts of the process have their own set of functions that will run independently. I’m writing my Lambda functions using C#/.NET Core 2.0 (I started with Go initially but I’ve used C# before and find that I personally prefer it).
So far I think the development process has been good. Deployment has been okay, and I’ve found it easier to deploy my .NET functions vs the old Go functions.
Logging/debugging is a bit of a chore but not awful. Cloudwatch alarms are fairly useful for knowing when something critical happened, but you still have to dig in fairly deep to find the relevant error information.
Performance wise, Go was really fast even with the smallest amount of memory. Cold starts were around 1s and requests after that were around 30-40ms on average. C# is substantially slower unless you bump up the memory.
All that being said, overall I really enjoy developing Lambda functions and not having to worry about containers, etc., because I’m not a DevOps person. I would definitely use it again for other projects, and probably will.
So far I think the development process has been good. Deployment has been okay, and I’ve found it easier to deploy my .NET functions vs the old Go functions.
Logging/debugging is a bit of a chore but not awful. Cloudwatch alarms are fairly useful for knowing when something critical happened, but you still have to dig in fairly deep to find the relevant error information.
Performance wise, Go was really fast even with the smallest amount of memory. Cold starts were around 1s and requests after that were around 30-40ms on average. C# is substantially slower unless you bump up the memory.
All that being said, overall I really enjoy developing Lambda functions and not having to worry about containers, etc., because I’m not a DevOps person. I would definitely use it again for other projects, and probably will.
We've been using AWS Lambda function to asynchronously render PDFs, transform data or execute on different pipeline triggers with success for a few years now. The horizontal scaling that avoids having a pressured queue during high-traffic spikes is pretty nice.
Another case we've had success with is transforming and relaying webhooks from different services (relaying events into Datadog for services that don't have a native integrations) or acting on occasional webhooks that don't happen too often (creating a default set of labels on a new Github repo).
If you're doing anything asynchronously, it's pretty nice. Be aware that the amount of memory you allocate to your functions is directly proportional to the CPU amount (and price) you're given. There's also the caveat that invocations perform _at least once_ but not necessarily once. A few months ago we saw a significant increase in repeat lamdaba invocations/retries even when the original execution was successful.
Another case we've had success with is transforming and relaying webhooks from different services (relaying events into Datadog for services that don't have a native integrations) or acting on occasional webhooks that don't happen too often (creating a default set of labels on a new Github repo).
If you're doing anything asynchronously, it's pretty nice. Be aware that the amount of memory you allocate to your functions is directly proportional to the CPU amount (and price) you're given. There's also the caveat that invocations perform _at least once_ but not necessarily once. A few months ago we saw a significant increase in repeat lamdaba invocations/retries even when the original execution was successful.
I’m also curious about it. My friend who have used it told me it’s difficult to debug. Is it true?
The idea looks good on paper but in practice it is pretty pricey and impractical. You get free sandboxing, scaling, and stateless deployment out of the box but are charged by the second. It's really optimized for simple web apps (or microservices) - think backend for form processing or resizing a picture. Anything moderately complex e.g. python scientific software is impossible due to the 50MB compressed app size limit (including all your dependencies), if your app takes more than a couple seconds of compute time, it will burn a hole in your pocket.
Usage: Algorithmic trading, Alexa voice apps, Brokerage price matching
http://KloudTrader.com
Usage: Algorithmic trading, Alexa voice apps, Brokerage price matching
http://KloudTrader.com
50 MB size limit can be overcome if you upload the zip package to S3 first.
It's limited to 50MB on S3. You're thinking of a much lower limit.
The 50 MB limit on S3 is on the .zip/.jar upload but you can have up to 250MB if it fits inside a 50MB zip file [1].
[1] https://docs.aws.amazon.com/lambda/latest/dg/limits.html
[1] https://docs.aws.amazon.com/lambda/latest/dg/limits.html
Started using Lambda 2 years ago now and my thoughts of it have evolved over time. I was initially frustrated by it but a lot of the pain points have disappeared over time (especially no Python 3 support; now supports 3.6).
Some random thoughts and gotchas in no particular order:
1. Don't do your own deployments. Use Serverless.js (https://github.com/serverless/serverless). It's in JS but that doesn't lock you into the nodejs lambdas, it can deploy to any of the runtimes.
2. Use Zappa if you want to host a "serverless website". But frankly, serverless websites are only for hobby projects right now.
3. CPU vs Memory scaling sucks. You have to scale both at once; you can't do high CPU/low memory. This hurts the wallet for compute-intensive tasks.
4. 5-min hard timeout on all functions. Don't use Lambda for long-running tasks.
5. Low-CPU lambdas (128MB tier) are terrible at cold starts if you have lots of stuff to import (which WILL BE THE CASE for Zappa applications!). If a cold start exceeds your timeout (which defaults to 30 sec), your Lambda will never complete and never get out of cold start.
6. If you're playing with S3, PUTs will hurt your wallet. A common application of Lambda (the famed "resize images" example) has a workflow that looks like this: GET on API gateway, PUT to s3, triggers an event which fires a Lambda, lambda processes payload, PUTs result back to S3. If processing your payload is fast, the PUTs in this case could be the most expensive part of your flow.
7. X-Ray is atrociously confusing.
8. No cloudwatch metrics for memory usage. Had to do my own using log metrics. Dumb.
9. There's a cool versioning system for the code artifact, kinda like ECS has. It's really nice to use except that unlike ECS it doesn't have native lifecycle rules, which means you have to do your own cleanups. Dumb.
10. Lambda patterns can be implemented more cheaply and more efficiently using tasks queues and autoscaling EC2s. Amazon's "efficient" compute allocation is outweighed by the margins on Lambda pricing. As with other Amazon services, you're paying for the system to be managed.
11. Aurora Serverless is super interesting, but FWICT nowhere near ready for prime time.
12. API gateway is a solution looking for a problem. If you have to use it (eg. you must trigger your lambda through an http request and you don't want to run a web server), use it in proxy mode.
13. Golang lambdas are very promising. I want to try them out.
14. There's a native canary system in Lambda/APIGW nowadays. Also want to try it out. Had to roll my own before it existed and now too scared to touch the production system.
15. If you hit concurrency issues, your problem might not be load, it might be throughput. Is your DB slower than usual? Common pattern: Increase in Lambda requests hitting your DB, slows your DB down, which in turn slows your lambdas down, which causes you to hit concurrency limits. All these metrics are tightly related.
16. Your tests should not assume a Lambda environment unless your app actually depends on the lambda environment (it generally shouldn't).
17. Use this thing: https://github.com/lambci/docker-lambda - Among other things, you can use it to build binary libs/dependencies you want to run on Lambda (cairo for example).
18. Use this thing: https://github.com/spulec/moto -- And also this thing: https://github.com/localstack/localstack
19. Cloudwatch logs suck. If you want your app to be debuggable, make sure you know how to find logs for individually-triggered lambdas (eg. if you're thumbnailing, you should be able to trace the logs for that one thumbnail back into cloudwatch easily)
20. Use Sentry. https://sentry.io/ - You can use the Sentry extra context parameters to help auditing various things (lambda environment, aws api results you get at runtime, etc)
Some random thoughts and gotchas in no particular order:
1. Don't do your own deployments. Use Serverless.js (https://github.com/serverless/serverless). It's in JS but that doesn't lock you into the nodejs lambdas, it can deploy to any of the runtimes.
2. Use Zappa if you want to host a "serverless website". But frankly, serverless websites are only for hobby projects right now.
3. CPU vs Memory scaling sucks. You have to scale both at once; you can't do high CPU/low memory. This hurts the wallet for compute-intensive tasks.
4. 5-min hard timeout on all functions. Don't use Lambda for long-running tasks.
5. Low-CPU lambdas (128MB tier) are terrible at cold starts if you have lots of stuff to import (which WILL BE THE CASE for Zappa applications!). If a cold start exceeds your timeout (which defaults to 30 sec), your Lambda will never complete and never get out of cold start.
6. If you're playing with S3, PUTs will hurt your wallet. A common application of Lambda (the famed "resize images" example) has a workflow that looks like this: GET on API gateway, PUT to s3, triggers an event which fires a Lambda, lambda processes payload, PUTs result back to S3. If processing your payload is fast, the PUTs in this case could be the most expensive part of your flow.
7. X-Ray is atrociously confusing.
8. No cloudwatch metrics for memory usage. Had to do my own using log metrics. Dumb.
9. There's a cool versioning system for the code artifact, kinda like ECS has. It's really nice to use except that unlike ECS it doesn't have native lifecycle rules, which means you have to do your own cleanups. Dumb.
10. Lambda patterns can be implemented more cheaply and more efficiently using tasks queues and autoscaling EC2s. Amazon's "efficient" compute allocation is outweighed by the margins on Lambda pricing. As with other Amazon services, you're paying for the system to be managed.
11. Aurora Serverless is super interesting, but FWICT nowhere near ready for prime time.
12. API gateway is a solution looking for a problem. If you have to use it (eg. you must trigger your lambda through an http request and you don't want to run a web server), use it in proxy mode.
13. Golang lambdas are very promising. I want to try them out.
14. There's a native canary system in Lambda/APIGW nowadays. Also want to try it out. Had to roll my own before it existed and now too scared to touch the production system.
15. If you hit concurrency issues, your problem might not be load, it might be throughput. Is your DB slower than usual? Common pattern: Increase in Lambda requests hitting your DB, slows your DB down, which in turn slows your lambdas down, which causes you to hit concurrency limits. All these metrics are tightly related.
16. Your tests should not assume a Lambda environment unless your app actually depends on the lambda environment (it generally shouldn't).
17. Use this thing: https://github.com/lambci/docker-lambda - Among other things, you can use it to build binary libs/dependencies you want to run on Lambda (cairo for example).
18. Use this thing: https://github.com/spulec/moto -- And also this thing: https://github.com/localstack/localstack
19. Cloudwatch logs suck. If you want your app to be debuggable, make sure you know how to find logs for individually-triggered lambdas (eg. if you're thumbnailing, you should be able to trace the logs for that one thumbnail back into cloudwatch easily)
20. Use Sentry. https://sentry.io/ - You can use the Sentry extra context parameters to help auditing various things (lambda environment, aws api results you get at runtime, etc)
That's a great list of gotchas, could you specify some main reasons why you'd still choose lambda with so many small issues vs "traditional" hosting?
It's not either-or. I wouldn't replace "traditional" hosting by Lambda, but among other things it is very suitable for the following use cases:
- You have a mostly static site that needs a couple of dynamic views (eg. comment or payment processing), but the site doesn't get enough traffic to put a webserver behind it.
- You are continuously processing sub-5min tasks and don't want to have to be doing the up/down scaling yourself and you're willing to pay a premium to have the scaling be managed.
- You want an easy way to run a piece of code from various AWS triggers without having to set up webhooks in between.
- Your next employer is a buzzword vulture and you have an idea for a serverless virtual reality blockchain.
- You have a mostly static site that needs a couple of dynamic views (eg. comment or payment processing), but the site doesn't get enough traffic to put a webserver behind it.
- You are continuously processing sub-5min tasks and don't want to have to be doing the up/down scaling yourself and you're willing to pay a premium to have the scaling be managed.
- You want an easy way to run a piece of code from various AWS triggers without having to set up webhooks in between.
- Your next employer is a buzzword vulture and you have an idea for a serverless virtual reality blockchain.
I used API Gateway and Lambda to serve 20M API calls last month on ipdata.co for a little over $100.
Something to note, latencies decrease with increased usage due to fewer cold starts. I've seen latencies as low as 17ms in our busiest region with ~40ms in less busy regions.
Something to note, latencies decrease with increased usage due to fewer cold starts. I've seen latencies as low as 17ms in our busiest region with ~40ms in less busy regions.
I never really understood why people would want to build "applications" on top of Lambda. I have always been under the impression that the primary function of Lambda is to assist in programming the AWS environment (in conjunction with SNS and SQS).
Expensive, especially if you are utilizing API Gateway. Otherwise, it's amazing.
I gave it a shot because my api was fairly simple and I really did not want to go through the hassle of setting up a server. The overhead turned out to be massive, like half a second per query. So I ditched it for node.
Running a pretty simple app on Lambda: take DMARC reports, parse them, ship them off to somewhere useful. Same app deploys to Lambda+SES and Google App Engine.
Lambda app: just keeps chugging. GAE app: perpetual disaster.
Lambda app: just keeps chugging. GAE app: perpetual disaster.
Thanks!