AWS Service Operator for Kubernetes Now Available(aws.amazon.com)
aws.amazon.com
AWS Service Operator for Kubernetes Now Available
https://aws.amazon.com/blogs/opensource/aws-service-operator-kubernetes-available/
55 comments
https://github.com/jtblin/kube2iam is probably what you're looking for, it uses iptables to allow/disallow pods requests to the ec2 metadata service based on kubernetes annotations
in fact, if you check out the source (located here: https://github.com/awslabs/aws-service-operator) it's recommended to use kube2iam
edit: haven't fully read the article yet but if the operator supports managing IAM roles thru a CRD you could potentially create the role and attach it via annotation in one go.
double edit: looks like IAM roles aren't directly supported yet, the following is what appears to be supported:
- cloudformation templates
- dynamodb
- s3
- sns subscriptions and topics
- sqs queues
- ecr repos
in fact, if you check out the source (located here: https://github.com/awslabs/aws-service-operator) it's recommended to use kube2iam
edit: haven't fully read the article yet but if the operator supports managing IAM roles thru a CRD you could potentially create the role and attach it via annotation in one go.
double edit: looks like IAM roles aren't directly supported yet, the following is what appears to be supported:
- cloudformation templates
- dynamodb
- s3
- sns subscriptions and topics
- sqs queues
- ecr repos
Correct, they are on the roadmap, I've been waffling on the implementation because this could open security issues. I'm happy to say we'll at least be able to use k8s RBAC to gate who can get, list, create, update and delete the Roles but your security posture from the node perspective still will need to gate what the pods could assume. https://github.com/awslabs/aws-service-operator/issues/58 https://github.com/awslabs/aws-service-operator/issues/59 are the issues if you'd like to add any extra notes or check out the potential implementation.
I understand that both kube2iam and kiam exist and they're both fine. I am interested in Amazon/EKS directly supporting something. Hopefully we will see something soon via CRD or otherwise.
If I still need to create an IAM role externally for the pod granting it access to the different resources.. Might as well create the resources themselves as well?
We've been using kube2iam for this for a couple of years.
You just create a role give it an assume role policy that allows the node to assume it. Then annotate your pod w/ the role arn. When they make a call to get their instance profile you get the role instead.
It's a little annoying in that your pod code thinks its making a metadata call (which is super super fast), but what is actually happening is kube2iam intercepted that and will make a sts:assumerole call... which takes forever. So people just need to set their timeout a little higher than normal.
https://github.com/jtblin/kube2iam
You just create a role give it an assume role policy that allows the node to assume it. Then annotate your pod w/ the role arn. When they make a call to get their instance profile you get the role instead.
It's a little annoying in that your pod code thinks its making a metadata call (which is super super fast), but what is actually happening is kube2iam intercepted that and will make a sts:assumerole call... which takes forever. So people just need to set their timeout a little higher than normal.
https://github.com/jtblin/kube2iam
This is something we’ve heard a lot and are working together with the Kubernetes community on. We’ll have more to announce on this one soon
Full disclosure: I work on EKS at AWS
Full disclosure: I work on EKS at AWS
I too am excited to hear more. The necessity for kube2iam and kaim are one of the major blockers that have prevented us from considering EKS for a client, the major sticking point being that they move the implementation of shipping security material around from the AWS to the customer side of the shared responsibility model.
This is exciting to hear. I look forward to hearing more.
AWS always launches services half-baked - I actually kinda love them for it. It's not always polished, but it's usually good enough to help me get stuff done.
I'm sure we'll see much tighter integration over time.
I'm sure we'll see much tighter integration over time.
I bet you could register the containers as on-prem instances with the Systems Manager agent, though it's kind of a pain. It gives you a shim into IAM from non-AWS machines.
(I work at AWS SSM, but not directly on the on-prem featureset.)
(I work at AWS SSM, but not directly on the on-prem featureset.)
For me, this is the most exciting thing AWS has launched in a while!
Just an attempt to de-comiditize cloud infra .. what we need is a cloud agnostic solution built by someone other than aws/azure/goog
What's wrong with them ? Three is enough competition.
Cost is what is wrong. They will form a cartel if there are no competing solutions that reign them in.
What terminal theme was that in the screenshot? I saw command icons, so it’s a Mac, but how do you make iterm look like that?
This is absolutely awesome. I've thought about doing this a couple of times, as a abstraction layer on top of different clouds, but this is really cool... It's also the first time that I think we have really seen AWS really contribute something to the K8s ecosystem they do lots of good work at the CNCF that is interesting and innovative. (EKS is not as capable as GKS or AKS, and even things like HPA only recently are enabled).
Does this work benefit those who are not using AWS?
It's great to see Kubernetes being integrated more tightly with the AWS ecosystem. If now all cloud providers open sourced their MySQL/PostgreSQL forks...
Are there any good docker compose or helm files out there right now that would work for a nice production ready MySQL or PostgreSQL dB?
It seems like k8s has everything you would need to have the redundant data sources, failover, and point in time recovery options that cloudsql or auroradb have.
It seems like k8s has everything you would need to have the redundant data sources, failover, and point in time recovery options that cloudsql or auroradb have.
The open source database vitess is pretty much designed to be a cloud native production mysql and there are helm charts for it (https://github.com/vitessio/vitess/tree/master/helm) an operator based on the the metacontrollers (https://github.com/vitessio/vitess-operator) and a company called planetscale thats making an operator that has even more features then those two (I work for planetscale and have been building our operator )
I am aware of Crunchy Data doing PostgreSQL-on-Kubernetes:
https://www.crunchydata.com/products/crunchy-postgresql-for-...
https://www.crunchydata.com/products/crunchy-postgresql-for-...
We run the official Postgres docker image as a StatfeulSet. Very easy to setup, and we haven’t had any issues with it.
Is there something equivalent for GCP? AFAIK you control load balancers via Ingress objects but that's about it in terms of integration, right?
GCP has service catalog[1] that can interface to various GCP services (Spanner, cloud sql, pub sub, etc.).
Service catalog is based on the open service broker spec.
[1] https://cloud.google.com/kubernetes-engine/docs/how-to/add-o...
Service catalog is based on the open service broker spec.
[1] https://cloud.google.com/kubernetes-engine/docs/how-to/add-o...
Amazon has had that for a while too, but I just learned about it recently, see [1]. I wonder if this is doing the same thing under the hood, or if it is a competing project within Amazon. Regardless, something like this is sorely needed for making infrastructure changes deployable along with application changes.
1. https://aws.amazon.com/blogs/opensource/provision-aws-servic...
1. https://aws.amazon.com/blogs/opensource/provision-aws-servic...
The difference between "meet customers at a Kubernetes CRD" and "meet customers at an OSBAPI broker" is small enough that it's not worth paying strategy tax to crimp the latter, especially since these customer groups will be overlapping but not equivalent sets (this is how enterprise software grows so vast).
Rephrasing: AWS are smart to have a bob each way.
Rephrasing: AWS are smart to have a bob each way.
The cloud fight of 2019-2020: AWS vs GCP in the Kubernetes arena. Curious to see who's going to win, of if it's going to be a tie.
Jokes apart: GCP got a head start in containers thanks to Kubernetes; AWS realized it and tried to catch up. Dominating the space will have huge consequences down the road.
My humble view is that whoever starts a RedHat-like service (with support, and SLAs, and enterprise services) on top of Kubernetes, might get the upper hand. Having built Kubernetes might not be enough for GCP to maintain the lead.
Jokes apart: GCP got a head start in containers thanks to Kubernetes; AWS realized it and tried to catch up. Dominating the space will have huge consequences down the road.
My humble view is that whoever starts a RedHat-like service (with support, and SLAs, and enterprise services) on top of Kubernetes, might get the upper hand. Having built Kubernetes might not be enough for GCP to maintain the lead.
> My humble view is that whoever starts a RedHat-like service (with support, and SLAs, and enterprise services) on top of Kubernetes, might get the upper hand.
Does Red Hat count as Red Hat-like? Because they've had OpenShift Origin for several years now.
Does Red Hat count as Red Hat-like? Because they've had OpenShift Origin for several years now.
As far as I know the container fights are done Kubernetes won even AWS admits it, now it's all about serverless technologies.
For serverless technologies to "win" they have to solve the "cold start" problem. AWS likes to pitch Lambdas as an easy mobile backed, but if you need to talk to a DB (which most mobile backends do) then you'll want to put your Lambda in a VPC, which makes cold starts on the order of 5-10 seconds, which is a deal breaker for most synchronous APIs.
I don't understand why AWS or GCP haven't added "pre-warming" requests to their cloud functions, similar to App Engine.
I don't understand why AWS or GCP haven't added "pre-warming" requests to their cloud functions, similar to App Engine.
It kind of makes sense, but mostly from a marketing and planning standpoint. If you need pre-warming, that probably means you know enough about lambda (and AWS) to know that you need pre-warming. Setting it up via a Cloudwatch event is easy as pie.
Think about the implications if they added a button to the lambda console to "pre-warm". There are two options: (1) set up the cloudwatch event for you (which is a similar pattern we've seen AWS use for things like DynamoDB table autoscaling), or (2) have some other internal system which can keep them warm.
Its easy to say "just do (1), it'd be so easy", but the issue is that it introduces a very weird cost pattern to lambda. Lambda isn't just billed per invocation, its billed essentially with time live. So if they auto-configure a cloudwatch event, lets say it sends an empty `{}` argument, they have no idea how long your function is designed to run given that input. Moreover, they don't even know that your function won't error with that input. So they've got this new feature and even they can't predict what it will do to your bill or system stability, given the fact that we're dealing with arbitrary code blobs.
The only option is (2). Now think about allocating engineering effort to this problem: as a manager, would you rather allocate a team to work on an extra complex scheduling parameter, or continue to improve the fundamental warm-up time for any function? Maybe both. But now you've got this extra parameter there which increases customer expectations and makes future scheduling work much more difficult.
Think about the implications if they added a button to the lambda console to "pre-warm". There are two options: (1) set up the cloudwatch event for you (which is a similar pattern we've seen AWS use for things like DynamoDB table autoscaling), or (2) have some other internal system which can keep them warm.
Its easy to say "just do (1), it'd be so easy", but the issue is that it introduces a very weird cost pattern to lambda. Lambda isn't just billed per invocation, its billed essentially with time live. So if they auto-configure a cloudwatch event, lets say it sends an empty `{}` argument, they have no idea how long your function is designed to run given that input. Moreover, they don't even know that your function won't error with that input. So they've got this new feature and even they can't predict what it will do to your bill or system stability, given the fact that we're dealing with arbitrary code blobs.
The only option is (2). Now think about allocating engineering effort to this problem: as a manager, would you rather allocate a team to work on an extra complex scheduling parameter, or continue to improve the fundamental warm-up time for any function? Maybe both. But now you've got this extra parameter there which increases customer expectations and makes future scheduling work much more difficult.
The keep-alive behaviour of Lambda is essentially a subsidy. I suspect that AWS have paid a lot of attention to ways to cut it down, or are at least hoping the they can drive down the idling cost without anyone noticing too much of a performance hit.
> For serverless technologies to "win" they have to solve the "cold start" problem.
I've said here and elsewhere before that autoscaling is easy to say and hard to do.
We keep looking to autoscalers to divine our economic preferences, which they cannot do for us. What's been missing is the ability to explicitly trade off latency for expense.
The best you can do is to a) attack startup time any how, any way possible, b) react sanely to unexpected traffic changes, c) make reasonable forecasts and d) explicitly tune cost of idleness vs cost of delay vs probability of delay. These help, but the problem will never fully go away.
(Unless you've discovered an escape hatch from either of causality or integral calculus. If you have, please share it with the class.)
I've said here and elsewhere before that autoscaling is easy to say and hard to do.
We keep looking to autoscalers to divine our economic preferences, which they cannot do for us. What's been missing is the ability to explicitly trade off latency for expense.
The best you can do is to a) attack startup time any how, any way possible, b) react sanely to unexpected traffic changes, c) make reasonable forecasts and d) explicitly tune cost of idleness vs cost of delay vs probability of delay. These help, but the problem will never fully go away.
(Unless you've discovered an escape hatch from either of causality or integral calculus. If you have, please share it with the class.)
I think this is why AWS promotes DynamoDB so much, because it's one of the few datastores that work sanely with non-VPC lambdas.
Or you can use serverless asynchronously only.
Have a low latency container based API with min replicas and auto-scale, almost like an atomic CRUD API. Move as much to async serverless which is triggered on events.
Have a low latency container based API with min replicas and auto-scale, almost like an atomic CRUD API. Move as much to async serverless which is triggered on events.
Hasn't AppSync already solved that problem?
why the need to put lambdas in a VPC in order to hit a DB?
A VPC in AWS is essentially a virtual datacenter.
For many years now, essentially all AWS services are tied to a VPC.
Each account gets 5 VPCs per region, by default.
Whether you use RDS or EC2 to setup a database server, it will be tied to a VPC for networking isolation purposes.
As such you then would need the Lambda in the VPC, or to allow public internet access to the database.
The point is pretty moot though, because you can schedule Cloudwatch Events every 4 minutes to keep a lambda warm, if necessary.
Frameworks like Zappa even do this for you automatically.
For many years now, essentially all AWS services are tied to a VPC.
Each account gets 5 VPCs per region, by default.
Whether you use RDS or EC2 to setup a database server, it will be tied to a VPC for networking isolation purposes.
As such you then would need the Lambda in the VPC, or to allow public internet access to the database.
The point is pretty moot though, because you can schedule Cloudwatch Events every 4 minutes to keep a lambda warm, if necessary.
Frameworks like Zappa even do this for you automatically.
> The point is pretty moot though, because you can schedule Cloudwatch Events every 4 minutes to keep a lambda warm, if necessary.
I encourage you to read this article, https://theburningmonk.com/2018/01/im-afraid-youre-thinking-... , because if you're running a web API with Lambdas, keeping one instance warm with the "cloudwatch event every 4 minutes" trick will most definitely not solve your cold start issues.
I encourage you to read this article, https://theburningmonk.com/2018/01/im-afraid-youre-thinking-... , because if you're running a web API with Lambdas, keeping one instance warm with the "cloudwatch event every 4 minutes" trick will most definitely not solve your cold start issues.
> I think many still repeat the "conventional wisdom" about the cold start, and never get past that point.
First comment on your article nails it. At the end of the day lambdas scheduling is a black box. People have deduced certain behavior, but AWS is explicit about not relying on undocumented behavior.
I would be loath to recommend lambda for any application where business performance is sensitive to the services latencies.
First comment on your article nails it. At the end of the day lambdas scheduling is a black box. People have deduced certain behavior, but AWS is explicit about not relying on undocumented behavior.
I would be loath to recommend lambda for any application where business performance is sensitive to the services latencies.
It is somewhat arbitrary though, isn't it? If AWS adds the ability to use security groups without a VPC, a lot of these issues go away wrt vpc or siloed infrastructure limits (cold starts).
There are still reasons to be in a private network - Being "one typo away" from exposing your services/db to the world is scary. But that seems like a solveable problem as well...
There are still reasons to be in a private network - Being "one typo away" from exposing your services/db to the world is scary. But that seems like a solveable problem as well...
[deleted]
I think to do otherwise you'd have to have the DB open to the public internet.
Because the DB doesn't have a public IP, which is very good.
Do you mean OpenShift?
https://www.openshift.com
https://www.openshift.com
I think that is heptio
Why didn't they launch with RDS support? Seems like a no brainer.
Great question, it's a little more complicated than one might think at first. In trying to build a "batteries included" experience I'd need to have per-view into your cluster and what VPC, Subnet and AZ you are running in I don't want to make this a configuration option so I need to build out a way to collect this information dynamically so that I can make sure we create DB subnets for the RDS to provision into. Then I need to configure depending on the engine (pg, mysql etc) the port and security group configuration. All in all the CFT and is more complicated and with the way the resources are code generated it requires heavy customization. All that being said it is well up on the top of my list to implement. Also always interested in letting other folks come and contribute if they feel inclined. :)
Ah, maybe I meant "it's a no brainer from a user perspective". Thanks for the explanation.
:) No worries, if you have other services that you find would be really useful please file issues https://github.com/awslabs/aws-service-operator/issues
I am curious if AWS has any plans to build an IAM integration for K8s that provides IAM credentials/roles directly to pods. An integration through EKS or K8s directly would make interacting with AWS resources very easy.
Being able to authenticate to the K8s cluster using https://github.com/kubernetes-sigs/aws-iam-authenticator is nice, but it doesn't help give pods IAM roles.