Pyinfra automates infrastructure super fast at scale(pyinfra.com)
pyinfra.com
Pyinfra automates infrastructure super fast at scale
https://pyinfra.com/
37 comments
I use pyinfra for my personal projects and I love it. Compared to Ansible it's a breath of fresh air. Ansible is very straightforward to start with, but quickly devolves whenever you step off the beaten path, and you end up with a tangled mess of unintelligible YAML after a while. Pyinfra is just Python. If you can read Python, you can understand what the deployment module is doing.
Hey all, I (& many amazing contributors) built pyinfra! Crazy to see it on the homepage I’ll do my best to answer any questions!
Does pyinfra play nicely with devices that provide an SSH interface, but aren't traditional Unix boxes?
I use Ansible currently to manage the config on a Ubiquiti EdgeRouter. Using actual Python would be even better.
I use Ansible currently to manage the config on a Ubiquiti EdgeRouter. Using actual Python would be even better.
Yes! I know of people that have connected to systems without any traditional shell/unix stuff. You can change the shell (or have none) used if needed using the global `_shell_executable` global arg (https://docs.pyinfra.com/en/2.x/arguments.html#shell-control...).
Most operations rely on various Linux/similar tools but the `server.shell` operation plus shell flag above should get you connecting and executing commands. Please do reach out if this doesn’t work for your setup!
Most operations rely on various Linux/similar tools but the `server.shell` operation plus shell flag above should get you connecting and executing commands. Please do reach out if this doesn’t work for your setup!
Interested in the same usecase, but for Mikrotik routers.
Late reply but I'm curious if the focus is idempotent execution (like ansible) or desired state (like puppet).
I'm guessing the former since it's agentless.
I'm guessing the former since it's agentless.
Read a few pages I gave up. May I ask
- what is its use case?
- security
- charge model
- self hosting
Sorry if miss a faq on the site ?
I haven't used this for large deployments but I use it for my personal server and it works perfectly. Almost everything is built in and I can easily write my own custom operations when I need them. Documentation is good and the operations are well designed.
Only downside is I couldn't make it work with my SSH agent, but that might be a problem with Paramiko and not Pyinfra.
Only downside is I couldn't make it work with my SSH agent, but that might be a problem with Paramiko and not Pyinfra.
> Only downside is I couldn't make it work with my SSH agent, but that might be a problem with Paramiko and not Pyinfra.
Interested to hear more about this, might be something that can be added to paramiko (or on top of).
Interested to hear more about this, might be something that can be added to paramiko (or on top of).
That the tool i am looking for a while. I used fabric before but it's too old and limited, no longer maintained.
Plumini is focused on cloud, not on dedicated or bare metal servers.
Why I don't like Ansible is : YAML is a bitch to maintain and python is much more dynamic and flexible.
So this is the real thing. I am going to use it for my DevOps life.
Keep it up!!
So this is the real thing. I am going to use it for my DevOps life.
Keep it up!!
Does this have any advantages over Ansible? It seems to have some config management, but not as robust or well documented. Ansible also does parallel remote execution.
Yes.
- you use an actual programming language (which you're likely to already know) instead of desperately figuring out how to simulate a for loop in yaml or to replace a part of a string, or whatnot
- you have all Python static analysis tools at your disposal. mypy/pylint etc can make your deploys less error prone
- you can easily implement a custom DSL in python, so deploys ends up more declarative and with less copy pasting
- it's easy to implement custom arbitrary operations (since you're working in python) -- and again you can reuse variables, etc instead of having to use some weird templating or pass them in command line
- you use an actual programming language (which you're likely to already know) instead of desperately figuring out how to simulate a for loop in yaml or to replace a part of a string, or whatnot
- you have all Python static analysis tools at your disposal. mypy/pylint etc can make your deploys less error prone
- you can easily implement a custom DSL in python, so deploys ends up more declarative and with less copy pasting
- it's easy to implement custom arbitrary operations (since you're working in python) -- and again you can reuse variables, etc instead of having to use some weird templating or pass them in command line
At what point are you just re-inventing what Ansible already does if you're going down the DSL path? I sometimes feel like I'm in the minority, but I truly do like Ansible and if you slap Mitogen[1] in front of it makes the whole process even more enjoyable!
I'm honestly trying to understand whether this hating on Ansible is more of a bandwagon thing (similar to how people shit on the Python dependency management ecosystem) and stems from having bad experiences that have completely turned them off of it.
Honest questions in response to your points:
1. What are you deploying where you would need static analysis?
2. If you did end up writing your own DSL on top of Python at what point would you concede that it's probably better to use someone's else creation as opposed to forever iterating on your own thing for the sake of using your own thing?
3. Why would you want "custom arbitrary operations" and how custom are they that Ansible and Jinja2 can't do that for you in usage patterns that are far more common than your own creation would be?
---
[1]: https://mitogen.networkgenomics.com/ansible_detailed.html
I'm honestly trying to understand whether this hating on Ansible is more of a bandwagon thing (similar to how people shit on the Python dependency management ecosystem) and stems from having bad experiences that have completely turned them off of it.
Honest questions in response to your points:
1. What are you deploying where you would need static analysis?
2. If you did end up writing your own DSL on top of Python at what point would you concede that it's probably better to use someone's else creation as opposed to forever iterating on your own thing for the sake of using your own thing?
3. Why would you want "custom arbitrary operations" and how custom are they that Ansible and Jinja2 can't do that for you in usage patterns that are far more common than your own creation would be?
---
[1]: https://mitogen.networkgenomics.com/ansible_detailed.html
One example where I personally ran into trouble with ansible was in dealing with SSL certificates. In production we use letsencrypt to generate SSL certificates as you'd expect, but for testing we were using self-signed certificates. This was surprisingly complicated under the ansible model, basically I ended up writing a reusable playbook that worked more or less as a function, and running a hook at the end to actually create the keys. Due to the specifics of our app depending on what kind you needed the actual certificate paths got the certificates were different.
So I ended up doing some crazy hacks passing around state instead of just writing a function.
So I ended up doing some crazy hacks passing around state instead of just writing a function.
Feels as though perhaps this should have dropped from playbook into a module?
https://docs.ansible.com/ansible/latest/dev_guide/developing...
Then just writing a function.
https://docs.ansible.com/ansible/latest/dev_guide/developing...
Then just writing a function.
It should have at least become a role that could be called in the different contexts of the environment it was targetting. Ansible's real power is in the hierarchical inventory, which can be used to efficiently describe the state of all of the things (I use it for everything from cloud things to specific machine things).
Sure, but there's something to be said for locality of behavior.
Does mitogen really only support up to Ansible 2.9? Thats a couple years old isn't it? I was really excited for it until I saw that :(
I'm on 'ansible==5.5.0' / 'ansible-core==2.12.3' and am using Mitogen with no issues whatsoever.
Have releases up to 2.13
https://github.com/willhallonline/docker-ansible
Still multiple major releases behind but still
Also I wonder how often the new releases are used (market share?) since so many people moved to cloud/k8s
https://github.com/willhallonline/docker-ansible
Still multiple major releases behind but still
Also I wonder how often the new releases are used (market share?) since so many people moved to cloud/k8s
> At what point are you just re-inventing what Ansible already does if you're going down the DSL path
For one, YAML can only emulate one kind of DSL: a declarative language. You have no advanced operations on top of YAML without using modern YAMLv3 syntax which is honestly mind boggling to read at times. Python is far more expressive and has tooling that can strengthen your operational resilience.
> I'm honestly trying to understand whether this hating on Ansible is more of a bandwagon thing
I've heard this expression echoed commonly when ones favorite tooling is under critique. Nobody hates Ansible, people do have experience that's taught them not to use something. Whether you feel your case overlaps with theirs is an exercise that pithy internet opinions can't sort out for you.
> What are you deploying where you would need static analysis?
Anything sufficiently dynamic. Static analysis is for making things less error prone. There's still all the same runtime problems, but at least there's an expressive path to fixing the runtime problems.
> 2. If you did end up writing your own DSL on top of Python at what point would you concede that it's probably better to use someone's else creation as opposed to forever iterating on your own thing for the sake of using your own thing?
That's a value judgement as to whether the person you're speaking to feels comfortable writing and maintaining a DSL. In my own experience, I've written things on my own that I had full control over that were sufficiently narrowly defined and I was able to move fast because of those things. I've also sat around waiting for features I'll never get or waiting for someone else's use case to become mine without avail because someone refused to "reinvent the wheel" without realizing the wheel they think exists isn't real.
> Why would you want "custom arbitrary operations" and how custom are they that Ansible and Jinja2 can't do that for you in usage patterns that are far more common than your own creation would be?
The custom orchestrators I've written were compensating for software that is deeply entrenched in it's infra and vice versa. Jinja buys me nothing that writing in Python or Go can, and Ansible is providing facilities the languages already give me. Ansible, in this case, is very much there to serve Jinja.
Of course, that's not to say that either way is wrong. Sometimes it's appropriate to use Ansible. Other times it's appropriate to write your own. Even in the latter decision, by doing so, it affords learnings about what you'll need beneath the hood so you can make educated decisions if you roll back to the former.
For one, YAML can only emulate one kind of DSL: a declarative language. You have no advanced operations on top of YAML without using modern YAMLv3 syntax which is honestly mind boggling to read at times. Python is far more expressive and has tooling that can strengthen your operational resilience.
> I'm honestly trying to understand whether this hating on Ansible is more of a bandwagon thing
I've heard this expression echoed commonly when ones favorite tooling is under critique. Nobody hates Ansible, people do have experience that's taught them not to use something. Whether you feel your case overlaps with theirs is an exercise that pithy internet opinions can't sort out for you.
> What are you deploying where you would need static analysis?
Anything sufficiently dynamic. Static analysis is for making things less error prone. There's still all the same runtime problems, but at least there's an expressive path to fixing the runtime problems.
> 2. If you did end up writing your own DSL on top of Python at what point would you concede that it's probably better to use someone's else creation as opposed to forever iterating on your own thing for the sake of using your own thing?
That's a value judgement as to whether the person you're speaking to feels comfortable writing and maintaining a DSL. In my own experience, I've written things on my own that I had full control over that were sufficiently narrowly defined and I was able to move fast because of those things. I've also sat around waiting for features I'll never get or waiting for someone else's use case to become mine without avail because someone refused to "reinvent the wheel" without realizing the wheel they think exists isn't real.
> Why would you want "custom arbitrary operations" and how custom are they that Ansible and Jinja2 can't do that for you in usage patterns that are far more common than your own creation would be?
The custom orchestrators I've written were compensating for software that is deeply entrenched in it's infra and vice versa. Jinja buys me nothing that writing in Python or Go can, and Ansible is providing facilities the languages already give me. Ansible, in this case, is very much there to serve Jinja.
Of course, that's not to say that either way is wrong. Sometimes it's appropriate to use Ansible. Other times it's appropriate to write your own. Even in the latter decision, by doing so, it affords learnings about what you'll need beneath the hood so you can make educated decisions if you roll back to the former.
so pulumi[1]?
[1]: https://github.com/pulumi/pulumi
[1]: https://github.com/pulumi/pulumi
As much as I want to love pulumi, their ssh story seems missing. It feels like pulumi is dedicated towards managing cloud instances through cloud apis, while I'd want to manage bare metal instances through ssh (or a local daemon). I'd love correction on this.
no, you're right. Terraform/Pulumi are much more focused on provisioning resources, while Ansible has its sweet spot in configuring a machine once its running.
I guess one would be that it is python and not Ansible.
How is the performance with larger deploy scripts? Ansible really struggles with hundreds of hosts and for simple things like uploading a dozen files to each host. The benchmarks are promising but the test really only does four things. https://docs.pyinfra.com/en/2.x/performance.html
Way back when I started pyinfra my only goal was performance I recall turning a 40 minute ansible run on 500ish hosts into 5 minutes. Big caveat: anecdotal and pretty old information there.
It’s been too long since I did perf testing but I still believe pyinfra Will significantly outperform ansible for the same tasks, you’ll have to try it ;) (also if perf is not good please raise an issue perf is absolutely a feature I wish to maintain)
It’s been too long since I did perf testing but I still believe pyinfra Will significantly outperform ansible for the same tasks, you’ll have to try it ;) (also if perf is not good please raise an issue perf is absolutely a feature I wish to maintain)
What’s the difference between pyinfra and fabric? Both seem to be similar.
idempotence
How would you use pyinfra to manage a fleet of edge devices ? The main difference with a park of servers being that the edge devices are mounted on vehicles and therefore switching on and off with their own rhythm each.
We cutrently get an alert from each device when an update is needed, ssh into it, and docker-compose the latest services.
Nice question! I am also wondering how it helps for the IoT devices...
Up to now, What we found out is that Pull way makes more sense instead of Push way for the fleet of more than 500Ks devices in the field.
We use Ansible Pull, but would also considering of such a way through GitOps approach.
Not sure, if it would be also provided via Pyinfra to get such stuff via Pull way.
Does any one have any links to some good writes ups on this being used in a bigger environment?
Anyone want to post their experience using it here?
Anyone want to post their experience using it here?
I wrote pyinfra so am biased but we used it at my previous company on a fleet of roughly 1.5k physical boxes (including deploys spanning all of them at once) extensively, so it scales to that level well. We had probably 30 groups within that each with their own deployments as well and all tied together using a single Jenkins box.
I would love to test beyond that but haven’t had the opportunity sadly! Performance and scale is a headline feature so will always look to push this further.
I would love to test beyond that but haven’t had the opportunity sadly! Performance and scale is a headline feature so will always look to push this further.
There was more commentary about it when it was posted in 2020: https://news.ycombinator.com/item?id=23487178 (although I didn't go through them all looking for endorsements)
It seems like Pyinfra is sort of declarative in the same style as Ansible. Is this a correct assumption?
Pyinfra is great, simple and no DSL! just pure python
Show HN: pyinfra v2 - https://news.ycombinator.com/item?id=30999030 - April 2022 (2 comments)
Pyinfra v2.0 Released - https://news.ycombinator.com/item?id=30973976 - April 2022 (3 comments)
Show HN: Pyinfra v1.4 - https://news.ycombinator.com/item?id=26983266 - April 2021 (3 comments)
Pyinfra – automate infrastructure super fast at scale - https://news.ycombinator.com/item?id=23487178 - June 2020 (64 comments)
Pyinfra v0.2 - https://news.ycombinator.com/item?id=12956784 - Nov 2016 (2 comments)