Bastions on demand in an AWS VPC(theconsultingcto.com)
theconsultingcto.com
Bastions on demand in an AWS VPC
https://theconsultingcto.com/posts/bastions-on-demand/
31 comments
Also worth mentioning you can also tunnel with SSM, so it's possible to use plain SSH or do anything that you usually used to accomplish with bastion.
https://github.com/elpy1/ssh-over-ssm
https://github.com/elpy1/ssh-over-ssm
While it's possible to do port forwarding with SSM [0] I would encourage people to test it well.
I've tried using it for development, and it was a terrible experience. The connections would hang all the time for no reason. I ended up starting an SSH session (via SSM) and done the port forwarding via that...
[0] https://docs.aws.amazon.com/systems-manager/latest/userguide...
I've tried using it for development, and it was a terrible experience. The connections would hang all the time for no reason. I ended up starting an SSH session (via SSM) and done the port forwarding via that...
[0] https://docs.aws.amazon.com/systems-manager/latest/userguide...
Or you can run ssm agent on a fargate task for double the fun (just need to register as hybrid and remember to clean up (scheduled lambda works a treat for this)
This is a neat concept and made reading the article and thread worth it. Thanks.
Always glad to see one of our products mentioned :)
Huge shameless plug for the coolness that is Session Manager, I can only think of a few scenarios where you wouldn't just prefer it over the alternatives. If you haven't played with it (or other SSM stuff), you totally should! It's cool and useful and easy and more or less free.
Huge shameless plug for the coolness that is Session Manager, I can only think of a few scenarios where you wouldn't just prefer it over the alternatives. If you haven't played with it (or other SSM stuff), you totally should! It's cool and useful and easy and more or less free.
It was a total no-brainer once we discovered it. Really nice work.
If you're interested in more feedback, I'd be happy to share. As with many things AWS, there were some undocumented rougher edges that took some trial and error to figure. Although the end result was well worth it. Totally understand if you're not on HN to do that sort of thing, though.
If you're interested in more feedback, I'd be happy to share. As with many things AWS, there were some undocumented rougher edges that took some trial and error to figure. Although the end result was well worth it. Totally understand if you're not on HN to do that sort of thing, though.
Hey, I'm all ears. Just an engineer, but customer feedback is taken super seriously (albeit with a tragically long turnaround time - it's a surprisingly large organization with a lot of irons on a lot of fires). I'll totally pass on anything you have to say though.
e: or put an email in your profile and I'll even reach out!
e: or put an email in your profile and I'll even reach out!
I don't keep an email in my profile but you can find me here: [email protected]
Is there a way to use SSM without using the access keys? I feel using access keys is incredibly not secure because rotating keys is a hassle and people might not do them all the time.
Also can u restrict access to ssh through ssm to certain ips?
Maybe I might have missed these, so any help would be appreciated.
Also can u restrict access to ssh through ssm to certain ips?
Maybe I might have missed these, so any help would be appreciated.
> Is there a way to use SSM without using the access keys? I feel using access keys is incredibly not secure because rotating keys is a hassle and people might not do them all the time.
Technically there is, you can use federated login. Might not be very convenient, depending on your identity provider.
A solution I use, while not technically "not using access kys" is storing them in the system credential store with aws-vault [0]. Works on Windows, Linux and Mac. And you can combine this with multi factor auth.
> Also can u restrict access to ssh through ssm to certain ips?
Yes, with an IAM policy. The policy below requires connecting with an MFA and from a specific IP range. It only allows connecting to a specific instance.
Technically there is, you can use federated login. Might not be very convenient, depending on your identity provider.
A solution I use, while not technically "not using access kys" is storing them in the system credential store with aws-vault [0]. Works on Windows, Linux and Mac. And you can combine this with multi factor auth.
> Also can u restrict access to ssh through ssm to certain ips?
Yes, with an IAM policy. The policy below requires connecting with an MFA and from a specific IP range. It only allows connecting to a specific instance.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "ssm:StartSession",
"Resource": [
"arn:aws:ec2:eu-west-3:123123123123:instance/i-123123123123",
"arn:aws:ssm:eu-west-3::document/AWS-StartPortForwardingSession",
"arn:aws:ssm:eu-west-3::document/AWS-StartSSHSession"
],
"Condition": {
"IpAddress": {
"aws:SourceIp": "1.2.3.4/32"
},
"BoolIfExists": {
"aws:MultiFactorAuthPresent": "true"
}
}
}
]
}
I'd love to use SSM with other OSes (i.e. FreeBSD) than just Amazon Linux & Ubuntu which are supported.
You can! The agent can be installed on almost any linux flavor, and is open source on GitHub, so it can be compiled and installed on any EC2 machines (or on-prem machines) you want. (This might not be easy, I haven't done it myself)
If you can make a good case for it, open a ticket or make a stink on the forums and there's a chance an engineer can be tasked to add it to their list of Officially Supported Platforms. This is how eg raspbian got added, though I don't know if the program is still running for adding platforms as it's been a few years.
If you can make a good case for it, open a ticket or make a stink on the forums and there's a chance an engineer can be tasked to add it to their list of Officially Supported Platforms. This is how eg raspbian got added, though I don't know if the program is still running for adding platforms as it's been a few years.
Does this still work if folks are authenticated via federation/SAML instead of as IAM users?
Yep.
Downside is that it doubles access options when you have a mixed environment, i.e. multiple clouds and/or non-cloud servers. Other than that it's a great solution if IAM is your default user source.
You can do this with a VPN as well, no bastion required.
Seems to me that the bastion approach is a bit weird at this point; no node should be directly attached to the public internet (unless you enjoy babysitting one). You usually have an LB in front of it, then perhaps a CDN+WAF (like CloudFlare) in front of that. The same can be done for SSH and even RDP.
The concept that your bastion is 'more secure' than your other systems needs to go away, make all of the systems that secure to begin with if you want to allow human access and shells.
Seems to me that the bastion approach is a bit weird at this point; no node should be directly attached to the public internet (unless you enjoy babysitting one). You usually have an LB in front of it, then perhaps a CDN+WAF (like CloudFlare) in front of that. The same can be done for SSH and even RDP.
The concept that your bastion is 'more secure' than your other systems needs to go away, make all of the systems that secure to begin with if you want to allow human access and shells.
Agreed on this. AWS Client VPN is a decent (though admittedly a little more pricey) option: https://aws.amazon.com/vpn/pricing/. But then again you're paying for the managed service.
Yeah, the managed part does add to the cost (understandably) but if the half-managed approach of appliances works for you I think OpenVPN-AS is an option as well.
Some people are using wireguard containers on Fargate to do the same thing.
Some people are using wireguard containers on Fargate to do the same thing.
Do you happen to have a solid link / reference / GitHub repo for Wireguard on Fargate? That's exactly what I'm looking for!
OpenVPN is a hot mess, but is currently the most supported mechanism across platforms.
Background: I want to give my (100% global remote) development team network access to our AWS dev environment Aurora Postgresql, EFS / NFS, redis, microservices, etc. We already have a local env with docker-compose but need to debug and test in the shared cloud dev environment.
OpenVPN is a hot mess, but is currently the most supported mechanism across platforms.
Background: I want to give my (100% global remote) development team network access to our AWS dev environment Aurora Postgresql, EFS / NFS, redis, microservices, etc. We already have a local env with docker-compose but need to debug and test in the shared cloud dev environment.
Sadly, no. A lot of the available 'nice' containers require host access that isn't available on Fargate. The ones people tend to use on Fargate are pure user-space versions that don't work as well as others. We have evaluated a bunch of finds from blogs and public repositories but they all lack one way or another.
Most of our setups use OpenVPN via OpnSense on AWS, second most popular option is OpenVPN-AS with a paid license, third is AWS OpenVPN, because of the price tag.
A few test setups rely on a small EC2 instance per group (t3.small for example) with a single container and it's a bit quirky to automate, especially on large user groups.
This is our main issue with WG in production so far: while the technology seems totally fine, it's not at a point where you can smoothly roll out a 'service' and get going, there are too many hoops to jump through and too many duct-tape constructions to make it integrate. (somewhat ironic, considering OpenVPN)
Most of our setups use OpenVPN via OpnSense on AWS, second most popular option is OpenVPN-AS with a paid license, third is AWS OpenVPN, because of the price tag.
A few test setups rely on a small EC2 instance per group (t3.small for example) with a single container and it's a bit quirky to automate, especially on large user groups.
This is our main issue with WG in production so far: while the technology seems totally fine, it's not at a point where you can smoothly roll out a 'service' and get going, there are too many hoops to jump through and too many duct-tape constructions to make it integrate. (somewhat ironic, considering OpenVPN)
I looked into this some time ago and found that wireguard cannot run on fargate due to requiring kernel modifications and the userspace implementation[0] has similar blockers.
[0]: https://github.com/aws/containers-roadmap/issues/239
[0]: https://github.com/aws/containers-roadmap/issues/239
Also interested in this approach
You can do this simply with an ec2 instance and starting it with aws cli as needed .youre only billed while it runs
I manage a lot of ec2 instances on dozens of AWS accounts... I can't see a reason to use this particular method... Even if you want to spin up bastion on demand like this, why Fargate, why terraform?
In the simplest form, I'd go with AWS CloudFormation of an EC2 instance. Remove the additional step of an ECS cluster if you don't already have one. An EC2 instance can access cluster VMs, or other VPC objects just as easily. Can it's wrap this in AWS SAM or CloudFormation + a controller like GitLab or Jenkins...
The benefit here is that when you're done, you have an AWS CloudFormation stack to kill... No need for anyone to have teraform installed to find and remove all associated resources.
Of course, this is all if you can't use Systems manager or the CLI to achieve your goals
In the simplest form, I'd go with AWS CloudFormation of an EC2 instance. Remove the additional step of an ECS cluster if you don't already have one. An EC2 instance can access cluster VMs, or other VPC objects just as easily. Can it's wrap this in AWS SAM or CloudFormation + a controller like GitLab or Jenkins...
The benefit here is that when you're done, you have an AWS CloudFormation stack to kill... No need for anyone to have teraform installed to find and remove all associated resources.
Of course, this is all if you can't use Systems manager or the CLI to achieve your goals
> why Fargate
Because it's easy to throw containers at the problem. Coupled with savings plans this can be pretty cheap.
> why terraform
Because it's easier for most than CloudFormation, and has all the benefits of decomposable stacks just like CF.
Because it's easy to throw containers at the problem. Coupled with savings plans this can be pretty cheap.
> why terraform
Because it's easier for most than CloudFormation, and has all the benefits of decomposable stacks just like CF.
I misread the title as “Bostonians demand an AWS VPC”. I was very confused on why, and how.
I don’t currently understand the additional security from bastion hosts. Yes you can have them short lived. But if someone cracks the bastion it’s not very good for security.
I think it’s better with VPN plus SSM.
I think it’s better with VPN plus SSM.
Arbitrary command execution via IAM is terribly scary to me - which is why I don’t use AWS systems manager.
[1] https://docs.aws.amazon.com/systems-manager/latest/userguide...