AWS Lambda Supports Python 3.7(aws.amazon.com)
aws.amazon.com
AWS Lambda Supports Python 3.7
https://aws.amazon.com/about-aws/whats-new/2018/11/aws-lambda-supports-python-37/
20 comments
> why does it matter what version of python lambda supports (or any framework / language for that matter)?
Python 3.7 is a lot faster (30 - 40%) than Python 3.6 for certain types of functions.
And you can't (easily) just install Python 3.7 from another runtime, because it has a ton of dependencies and takes minutes to compile. So even if it's technically possible, it precludes a lot of use cases.
Python 3.7 is a lot faster (30 - 40%) than Python 3.6 for certain types of functions.
And you can't (easily) just install Python 3.7 from another runtime, because it has a ton of dependencies and takes minutes to compile. So even if it's technically possible, it precludes a lot of use cases.
I actually dealt with a similar problem a couple years ago, we were trying to get numpy on Lambda but there's C compilation that needs to happen. What I ended up doing was provisioning an EC2 instance with the same settings as the environment that Lambda uses, then building everything there, and shoving the binaries in the Lambda. It worked, I'm pretty sure the same thing would work for building Python.
AWS just released a solution to run any programming language on AWS Lambda: https://aws.amazon.com/blogs/aws/new-for-aws-lambda-use-any-...
Ok, so I take from this that the post about specifically Python 3.7 doesn't make much sense really (as a HN post).
It makes sense because this is an AWS supplied executable, where you just upload the .py files.
If you follow the "Run any language" approach, you need to bundle the python interpreter and any libraries required.
If you follow the "Run any language" approach, you need to bundle the python interpreter and any libraries required.
AWS provides the Python 3.7 runtime for you, so you don't need to bundle it. The post refers to runtimes that AWS doesn't provide, such as Haskell.
Genuine question... Does it support the size of a typical (for data, ml, image proc, etc) Python dependency bundle?
The deployment package size limit seems to be 250MB: https://docs.aws.amazon.com/lambda/latest/dg/limits.html
That's the limitation for data related worker tasks. The iron.io can be alternative (they offer 512 MB https://try.iron.io/pricing-worker-monthly/)
That would be OK for some ML applications if the framework itself would not be included in the deployment package.
There are workarounds for that, f.e. on startup loading code from an s3 bucket (https://hackernoon.com/exploring-the-aws-lambda-deployment-l...)
That workaround is for the 50MB limit on zip files, not for the 250MB limit :)
No, you get another 500 MB of tmp space you can use at runtime. This workaround does work.
I have a deployed lambda with your usual stack ( numpy sklearn scipy)
Did you manage to do that without pulling them from S3 (i.e., get them to fit in the 250MB limit)?
I'm using serverless-python-requirements (see the specific example https://github.com/UnitedIncome/serverless-python-requiremen...) - it uploads the libs zipped and unzips them on first invocation in the handler.
For a lot of data processing and indigestion in a commercial setting, the standard library is often enough.
Had a Lambda that parsed a file attached to an incoming email, and it worked very well. Even with complex packages like lxml, it usually isn't an issue. Seems like it might be tricky with some scientific packages or ML stuff though, but Lambdas aren't great for those workloads anyway, although the 15 minute runtime increase has helped.
Had a Lambda that parsed a file attached to an incoming email, and it worked very well. Even with complex packages like lxml, it usually isn't an issue. Seems like it might be tricky with some scientific packages or ML stuff though, but Lambdas aren't great for those workloads anyway, although the 15 minute runtime increase has helped.
Couldn’t you just pick the go runtime for example and have it execute the framework / language you want with minimal overhead given it will be a call between two processes running on same host, which you can keep running between invocations.