AWS Cloud Control API, a Uniform API to Access AWS and Third-Party Services(aws.amazon.com)
aws.amazon.com
AWS Cloud Control API, a Uniform API to Access AWS and Third-Party Services
https://aws.amazon.com/blogs/aws/announcing-aws-cloud-control-api/
77 comments
I don't know why hashicorp and pulumi are all smug about this. In one go it's destroyed half their moats.
I don't know anything about Pulumi, but for Terraform I'd say err no?
Terraform is declarative; this is imperative.
Terraform doesn't try to make a common CRUD interface to everything on AWS; that is the sole distinction between this and the pre-existing CLI.
Terraform is declarative; this is imperative.
Terraform doesn't try to make a common CRUD interface to everything on AWS; that is the sole distinction between this and the pre-existing CLI.
Cloud Control provides a direct API to the primitive functions that underpin CloudFormation resources. So, if you look closely at what it's doing (hint: "desired state" parameter), Cloud Control is declarative on the resource side.
I don't think that replaces other AWS resource schedulers like Terraform and Pulumi, but it could, uh, embrace them into a more CloudFormation-oriented ecosystem, which is clever.
I don't think that replaces other AWS resource schedulers like Terraform and Pulumi, but it could, uh, embrace them into a more CloudFormation-oriented ecosystem, which is clever.
Yeah or open the doors to competitors since it'd be far easier to get that piece done. Terraform would just become a planner and a half baked language
Pulumi lets me create custom providers, leverage providers for other clouds/ products, and build code-based abstractions. I see no reason why I'd stop using it.
Not really a moat, at least for HashiCorp. Consider that Pulumi was able to get off the ground by using Terraform providers, including HashiCorp's AWS provider, behind the scenes.
In addition to the other replies, Terraform has used ARM on Azure for a while now which is similar to this a unified API for all Azure Resource Management (hence ARM) and this hasn't caused any issues for them there.
Nobody who is using terraform or pulumi en masse is going to flip the switch to cloudformation because of this particular change.
People use Hashicorp and Pulumi not to use AWS by api but to provide options to move to other providers easily.
In most places I've been, Terraform was used more for documentation than mobility. Sure, you don't need to unlearn Cloudformation if you switch to GCP, but you have a ton of AWS-specific stuff in your TF config that's going to need to be ported to the appropriate counterparts in your next cloud. But having a documented change history in Git for (almost) everything was much more useful than theoretical mobility.
I think this is really similar to the "we use kubernetes so we don't get locked into AWS" argument; true in theory, never pans out in practice.
I'd hazard that the number of engineering orgs using a single Terraform "stack" (repository? domain? whatever) to manage resources across multiple providers is in the single-digit percentages.
The ability to transfer knowledge has non-zero benefit, but the reality is; that knowledge is 10% "Terraform & HCL", 90% "AWS Resource Configuration & Best Practices". 90% of it doesn't transfer, and most of the remaining 10% could be learned quickly enough.
Maybe it enables you to transfer CI/tooling between cloud providers should your org switch. Probably the most reasonable argument.
I think the core of what you're saying is actually right, for the same reasons that the kubernetes statement above is right; AWS is really becoming its own isolated world. Terraform on AWS makes less and less sense, as their in-house options like CF, CDK, SAM, etc evolve maturity beyond what TF is capable of. TF (and Pulumi) (like Kubernetes) is for everything else.
I'd hazard that the number of engineering orgs using a single Terraform "stack" (repository? domain? whatever) to manage resources across multiple providers is in the single-digit percentages.
The ability to transfer knowledge has non-zero benefit, but the reality is; that knowledge is 10% "Terraform & HCL", 90% "AWS Resource Configuration & Best Practices". 90% of it doesn't transfer, and most of the remaining 10% could be learned quickly enough.
Maybe it enables you to transfer CI/tooling between cloud providers should your org switch. Probably the most reasonable argument.
I think the core of what you're saying is actually right, for the same reasons that the kubernetes statement above is right; AWS is really becoming its own isolated world. Terraform on AWS makes less and less sense, as their in-house options like CF, CDK, SAM, etc evolve maturity beyond what TF is capable of. TF (and Pulumi) (like Kubernetes) is for everything else.
Personally I use Terraform because it's multi-provider not so I can switch, but so that I can use whatever I want concurrently.
I have no interest in using CloudFormation for the AWS parts, and then.. what, Terraform anyway for any other provider?
(Even if CF were on par, which IMO it isn't.)
I have no interest in using CloudFormation for the AWS parts, and then.. what, Terraform anyway for any other provider?
(Even if CF were on par, which IMO it isn't.)
Terraform has multi-cloud support but it is not cloud agnostic. You have to rewrite your terraform for the things that you want to move to another provider.
Moving to other providers is unfortunately rarely easy except for all but the most basic infrastructure.
Moving to other providers is unfortunately rarely easy except for all but the most basic infrastructure.
I think that the main usage pattern for a tool like terraform is be able to version control your config, and , on the way, apply the git flow/git ops to infra changes (infra as code).
I don't think the (theoretical) ability to move to another infrastructure provider is the main motivation to use those tools. It might be a supporting motivation. But definitely not the real reason why they're using it. Hardly anybody actually changes their infrastructure provider anyway
I'll honestly just say it outright: this is horrible advice.
People using this argument has clearly never needed to be multi-cloud. If you can ever, ever, ever, avoid being absolutely forced into this, you should. Being able to move fast on infrastructure has such big implications, and being multi-cloud basically means that you can only rely on an extremely minimal subset of features that completely align across the clouds (hint: the less managed stuff).
People using this argument has clearly never needed to be multi-cloud. If you can ever, ever, ever, avoid being absolutely forced into this, you should. Being able to move fast on infrastructure has such big implications, and being multi-cloud basically means that you can only rely on an extremely minimal subset of features that completely align across the clouds (hint: the less managed stuff).
If it only was so easy to move! Sure, one tool, one configuration format but it’s not so easy to simply change some configs and move.
Even going for the provider lowest common denominator implies maintaining different configurations.
Even going for the provider lowest common denominator implies maintaining different configurations.
You'd pretty much have to start from scratch switching between clouds even with Terraform. All the providers map pretty closely to the underlying nature of the resources defined in other clouds.
In what world does TF or pulumi code written for Aws runs on gcp or azure? I mean, maybe you can cook something with ts and pulumi but it is not realistic
[deleted]
They only released support for it for the legacy AWS CLI v1, but not for AWS CLI v2.
Released for AWS CLI v2 as well now
I feel like the more complex corners of the AWS resource model (e.g. offer-accept stuff between accounts, in TGW for example) are going to get over simplified behind this CRUDL API.
My use case is for mostly read-calls. I just tried it for DynamoDB Table, SQS Queue, SNS Topic, and EC2 Instance, Kinesis Stream. With these resources, only Kinesis supports List call. SQS Queue & SNS Topic does not even support READ. It might be useful for resource creation, but I would go nuts to discover what resource and what APIs are supported.
Yo more complexity !
They(AWS) should hire the UX team from "Hetzner Cloud Services" - sure it doesn't have the same features, but dammit it's super easy to navigate.
What does this change compared to the current status ? Terraform/Pulumi/AWS CDK already use AWS API. Is it just about having consistent API calls naming ?
lukehoban says it best in https://news.ycombinator.com/item?id=28711903 where he talks about the three things CloudFormation provides.
The Cloud Control API is an API to the resource model. It unlocks the ability for AWS teams to contribute to the resource model without having to worry about the rest of what Cloudformation does. As a service owner interested in contributing to other systems like TF and Pulumi, now I just have to contribute to one thing. In the medium/long term it's going to be a lot more efficient way to evolve how we all use AWS resources in our favorite Infrastructure-as-code systems.
Disclosure: Work at AWS (but not on the CloudFormation team)
The Cloud Control API is an API to the resource model. It unlocks the ability for AWS teams to contribute to the resource model without having to worry about the rest of what Cloudformation does. As a service owner interested in contributing to other systems like TF and Pulumi, now I just have to contribute to one thing. In the medium/long term it's going to be a lot more efficient way to evolve how we all use AWS resources in our favorite Infrastructure-as-code systems.
Disclosure: Work at AWS (but not on the CloudFormation team)
Hmm this would also be nice for ansible.
Interesting. One could see this being a way to operate cloud resources through a Kube API.
You should take a look at CrossPlane for that kind of thing. They have been doing it for a couple of years.
[deleted]
Luc van Donkersgoed wrote a blog post about inconsistent AWS APIs just a week ago [1]. Funny coincidence.
[1]: https://www.lastweekinaws.com/blog/how-aws-dumps-the-mental-...
[1]: https://www.lastweekinaws.com/blog/how-aws-dumps-the-mental-...
Pretty awesome. Now IaC tools like Pulumi will be always up-to-date.
It will be super easy to write alternatives to Pulumi.
Will it? Given how Pulumi can use Terraform providers, I'd say the hard work there isn't the connectors (which this change makes easy) but the framework and the orchestration bits. These aren't made any easier by AWS standardizing their APIs.
I wouldn't say super easy, but at least much easier than before.
Indeed! And we just released a new AWS Native provider which builds on top of this new AWS Cloud Control API: https://www.pulumi.com/blog/announcing-aws-native/.
I'm really excited about what Cloud Control API enables more broadly as well. I see this as a new high bar for cloud provider resource provisioning APIs, and it's an approach I hope to see more of across the industry going forward.
Previously, to provision and manage a resource, you would need to find the REST APIs for that specific resource, and define your own model for what set of REST API calls constitute a logical Create, Read, Update or Delete operation. This work was done and re-done in dozens or hundreds of cloud provisioning tools.
But AWS themselves had already built a model for doing this, and service teams at AWS were already defining their own well-defined resource model - the CloudFormation resource model for their services/features - it was just locked up behind CloudFormation templates as the only way to author and deploy resources using this resource model.
With Cloud Control API, all of the value of that resource model, and the work by teams at AWS to cleanly define that resource model, is now unlocked to be used by any sort of provisioning tool or service - not just CloudFormation templates.
At Pulumi, that has enabled us to build our new AWS Native provider on top of Cloud Control API, but it opens up tons of additional scenarios as well. Kudos to the AWS team for their work on this!
I'm really excited about what Cloud Control API enables more broadly as well. I see this as a new high bar for cloud provider resource provisioning APIs, and it's an approach I hope to see more of across the industry going forward.
Previously, to provision and manage a resource, you would need to find the REST APIs for that specific resource, and define your own model for what set of REST API calls constitute a logical Create, Read, Update or Delete operation. This work was done and re-done in dozens or hundreds of cloud provisioning tools.
But AWS themselves had already built a model for doing this, and service teams at AWS were already defining their own well-defined resource model - the CloudFormation resource model for their services/features - it was just locked up behind CloudFormation templates as the only way to author and deploy resources using this resource model.
With Cloud Control API, all of the value of that resource model, and the work by teams at AWS to cleanly define that resource model, is now unlocked to be used by any sort of provisioning tool or service - not just CloudFormation templates.
At Pulumi, that has enabled us to build our new AWS Native provider on top of Cloud Control API, but it opens up tons of additional scenarios as well. Kudos to the AWS team for their work on this!
Now if we could just use cfn-signal (or something like it) when setting up an EC2 auto-scaling group without having to create a Cloudformation stack behind the scenes as the Pulumi awsx package does.
Pardon my density, but isn't that the function of the health checks built into the ASG, which offers both EC2 and ELB health checking to take a bogus instance out of service?
Or do you mean "operation A --> ASG --> (wait $foo time for 'healthy') --> operation B" type thing?
Or do you mean "operation A --> ASG --> (wait $foo time for 'healthy') --> operation B" type thing?
Is a native provider planned for Cloudflare?
I'm really excited that Pulumi supports this from day one: https://www.pulumi.com/blog/announcing-aws-native.
Why:
Awscli loves its arguments. Several people are going to point me at autocompletions, tell me zsh has this really great feature where .. all fine. But why have this 'problem' to solve in the first place? I don't get it.
For non-interactive use, sure, I don't care how verbose it is, it's even a sort of self-documenting feature. Is that it? They just really want all interaction via the web console?
aws cloudcontrol delete-resource \
--type-name AWS::Lambda::Function \
--identifier ukjfq7sqG15LvfC30hwbRAMfR-96K3UNUCxNd9
aws cloudcontrol delete-resource \
AWS::Lambda::Function \
ukjfq7sqG15LvfC30hwbRAMfR-96K3UNUCxNd9
Awscli loves its arguments. Several people are going to point me at autocompletions, tell me zsh has this really great feature where .. all fine. But why have this 'problem' to solve in the first place? I don't get it.
For non-interactive use, sure, I don't care how verbose it is, it's even a sort of self-documenting feature. Is that it? They just really want all interaction via the web console?
All AWS CLI commands follow the format 'aws [service] [action] [flags...]'. To my knowledge, there are exactly zero instances where positional arguments are used outside of naming the service and action; its always flags.
Whether this is best or not is a different discussion; I'm simply pointing out the irony of suggesting the CLI API for a service designed to make their APIs more consistent, should be inconsistent with the other CLI APIs customers are used to.
Whether this is best or not is a different discussion; I'm simply pointing out the irony of suggesting the CLI API for a service designed to make their APIs more consistent, should be inconsistent with the other CLI APIs customers are used to.
Re: positional arguments, I was surprised recently to find this one (`outfile`) in the AppConfig API.
https://awscli.amazonaws.com/v2/documentation/api/latest/ref...
https://awscli.amazonaws.com/v2/documentation/api/latest/ref...
Yes I know, I was sort of using it (new CLI designed to be a better interface) to say why is it so.
Apparently I hold a minority opinion that's so offensively contrary it deserves downvotes though!
I maybe didn't make it clear enough, but I think it's somewhere between 'don't care either way' and 'great' for non-interactive/scripted use. I just don't like the UX at all for using it interactively, I find it extremely slow and cumbersome.
This looks to be a big improvement, but - to me - just leaves such easy obvious trivial stuff on the table.
Apparently I hold a minority opinion that's so offensively contrary it deserves downvotes though!
I maybe didn't make it clear enough, but I think it's somewhere between 'don't care either way' and 'great' for non-interactive/scripted use. I just don't like the UX at all for using it interactively, I find it extremely slow and cumbersome.
This looks to be a big improvement, but - to me - just leaves such easy obvious trivial stuff on the table.
The AWS CLI is old and for the most part auto-generated from the API definitions. I actually prefer the first example to the second one, but in general I’m happy that it’s consistent. Even if it’s sometimes consistently annoying (help, not —help).
Other than the flag name, what's other significance of this difference? The flag seems trivial to me. Warp it in a shell script seems sufficient.
What do you mean other than? I removed it, that is the (entire) difference.
> Warp it in a shell script
As I said, for non-interactive use of course I don't care. But trying to actually do/check/test anything with awscli interactively/'manually' is pretty painful IMO.
> Warp it in a shell script
As I said, for non-interactive use of course I don't care. But trying to actually do/check/test anything with awscli interactively/'manually' is pretty painful IMO.
I disagree here. The first example is much more self-describing and follows the same paradigm that the rest of the AWS CLI already follows. I'd prefer erbosity here rather than the AWS CLI trying to figure out what I'm doing.
I'm of the same opinion - I'd rather not have to consider the argument order, and instead just be able to append flags to an interactive command in any order I see fit as I'm refining queries.
And you no longer have to worry about the meanings of positional arguments changing between versions and potentially running the wrong command.
jeez write your own wrapper script then. I don't understand what this needs to be bikeshedded for.
That is a solved issue . It is the standard way of command line syntax and provides a common way of parsing command line for many OS. You can certainly do things “another” way but all it will do is break decades worth of work for standardization and good old fashioned common sense. Make things modular so you can expand in different ways.
Well that's not really true, or at least incomplete. GNU & BSD have different styles; I believe both do positional/unnamed arguments.
It's true that awscli goes for all --named=args currently, but that's not decades of it not happening in general, and I just commented on wishing it weren't the case here.
I think this is a great simplification/abstraction, and I don't see why it couldn't go further and just be `aws <crud> <type> <resource>`. But that's apparently highly disagreeable even as a mere thought or suggestion, so I'll shutup.
It's true that awscli goes for all --named=args currently, but that's not decades of it not happening in general, and I just commented on wishing it weren't the case here.
I think this is a great simplification/abstraction, and I don't see why it couldn't go further and just be `aws <crud> <type> <resource>`. But that's apparently highly disagreeable even as a mere thought or suggestion, so I'll shutup.
Such verbosity (and other bloat and complexity in how AWS does things) is why I could never get excited about S3, EC2 et seq. when these services were first announced. Rightly or wrongly, I believe it is because most of AWS was originally implemented in Java. Programming in Java requires a certain comfort with verbosity and clunkiness. That showed through in early AWS interfaces and documentation. Indeed, it has alway been painful to interact with manually.
Whatever language is used, there is a choice of either positional or named arguments.
Consistently using named arguments for a CLI to an API seems to be a good idea. Nothing to do with whether the API is written in Java.
All of the AWS CLI support input in JSON or YAML as well and will generate the skeleton of the call for you as well.
Consistently using named arguments for a CLI to an API seems to be a good idea. Nothing to do with whether the API is written in Java.
All of the AWS CLI support input in JSON or YAML as well and will generate the skeleton of the call for you as well.
There's an xkcd for that: https://xkcd.com/927/
Not sure why you are getting downvoted, this was the first thing that sprang to mind for me.
"Oh, yes, another option." On top of terraform, pulumi, CloudFormation, the AWS SDKs, the CLI, and the CDK.
I mean, I appreciate them trying to push things forward and realize all options aren't all equivalent, but where's the deprecation to make things simpler?
"Oh, yes, another option." On top of terraform, pulumi, CloudFormation, the AWS SDKs, the CLI, and the CDK.
I mean, I appreciate them trying to push things forward and realize all options aren't all equivalent, but where's the deprecation to make things simpler?
Just buy Pulumi already and have them do it the way it's supposed to be. It's ridiculous that AWS documentation usually contains dozens of error prone manual steps.
Sorry to hear you found errors in the doc. Please report error to AWS using the button on the top of the page. https://aws.amazon.com/premiumsupport/knowledge-center/send-...
I wonder how long before this becomes a standard like swagger. People will create a spec for a well-known endpoint and probably call it CURDL that allows APIs to be self documenting and automatically compatible with a general universal API client. No one will have to maintain their own clients any more and everyone will wonder why it took so long to think of this...
There's Smithy, which tries to help design API operations around a resource in a CRUDL-like fashion - I think that's what you're getting at? https://awslabs.github.io/smithy/
It is super interesting to me that they now have essentially a declarative API in CloudFormation and now this corresponding imperative version with the same format. I'm not quite sure why you would need the imperative version. Anyone have any ideas? Couldn't Pulumi have just compiled their code into CloudFormation and deployed it?
CloudFormation is really three separate things rolled up into one: (1) a resource model for AWS, (2) a deployment orchestration engine and (3) a syntax for specifying desired state as CloudFormation templates.
Tools that compile to CloudFormation templates offer a way to access (2) directly - to use alternative front-end syntaxes but to still deploy via the CloudFormation orchestration engine.
Tools like Pulumi (and Terraform) have their own deployment orchestration engines, which we believe offer many benefits - performance, secrets, components, transformations, aliases/refactoring, multi-cloud provisioning, and a lot more.
Cloud Control API lets us (and others like us) access (1) directly, without having to use (2) or (3), and thus being able to still offer the full set of benefits of provisioning via Pulumi, along with the full set of benefits of a well-defined AWS resource model.
Tools that compile to CloudFormation templates offer a way to access (2) directly - to use alternative front-end syntaxes but to still deploy via the CloudFormation orchestration engine.
Tools like Pulumi (and Terraform) have their own deployment orchestration engines, which we believe offer many benefits - performance, secrets, components, transformations, aliases/refactoring, multi-cloud provisioning, and a lot more.
Cloud Control API lets us (and others like us) access (1) directly, without having to use (2) or (3), and thus being able to still offer the full set of benefits of provisioning via Pulumi, along with the full set of benefits of a well-defined AWS resource model.
As a concrete example of what this enables, the Pulumi Automation API lets you embed IaC straight into a bigger program. Folks have used this to create infrastructure-oriented SaaS products, self-service portals, and new higher level abstraction frameworks and tools, for instance, often spanning multiple cloud resource types (AWS, Azure, Kubernetes, Cloudflare, others) -- https://www.pulumi.com/automation/. Transpiling to YAML and handing it over to CloudFormation is clumsy and wouldn't work for these cases among others.
Imperative version can fill more dynamic use cases, like spinning up entire servers in response to something (eg user wants to create a game server at the click of a button).
Not something you can easily do with the declarative version.
Not something you can easily do with the declarative version.
> Not something you can easily do with the declarative version.
Its something that you can do easily with the imperative API for CloudFormation plus the declarative description of the stacks in CloudFormation.
Its something that you can do easily with the imperative API for CloudFormation plus the declarative description of the stacks in CloudFormation.
That new game server could just be another cloudformation stack but I get your point.
I don’t understand how this changes something? Now the type of resource you want to create and many different bodies depending on that will be inside the CreateResource instead of whatever function the specific AWS API had.
The important difference is that the wait semantics are built in, meaning that each operation no longer needs custom code. This is a boon for things like Pulumi and Terraform providers.
I highly doubt that tools like Terraform/Pulumi will not have custom code for each type of resource. Each type still needs to have its own custom Body with specific data types, some mandatory, some optional. Even if TF supports Cloud Control as yet another resource, using it is going to be a huge mess of untyped/unchecked/undocumented bodies.
The bodies are specified in a schematised way, so all of this code can be generated. This is not hypothetical - Pulumi’s Kubernetes support, Azure (“nextgen”) support and GCP Native providers are all built like this.
It’s quite possible that the provided schema may need some augmentation for some resources (for example, semantic diff for IAM policies) to get the best plan-time experience, but this is a different world from having to hand write schema and CRUD logic for every resource type per the last 7 years of Terraform AWS support.
It’s quite possible that the provided schema may need some augmentation for some resources (for example, semantic diff for IAM policies) to get the best plan-time experience, but this is a different world from having to hand write schema and CRUD logic for every resource type per the last 7 years of Terraform AWS support.
Pulumi promises same-day support on their "Native" providers like this one, so I think they really don‘t.
Where is the sdk?
The only addition introduced here is the support for third-party CloudFormation resources. So the same AWS APIs can now be used to directly manipulate third party resources without using CloudFormation at all.