Ask HN: Would you know a “Python idempotent incremental update manager”?
8 comments
My feedback for those who helped me or are interested in following up :)
Pyinfra was the perfect match.
I have used their "operations" to pile up the actions that come with the next release:
- A new package (apt)
- an updtated pip dependency
- checking a file
- and docker-compose actions to update all containers
I was able define my "inventory" easily, which allows me to update my whole fleet with one single command. One caveat here, the timeout is not properly configurable with a ProxyJump. Discussion opened on stackoverflow.
Pyinfra idempotency and dry-run approaches allow me to run the update as many times as I want.
For the next release, would I need to change anything different, I will simply add a new operation. The beauty of it is that all operations are strictly executed in the order I define them, which mean that any devices that would not have been updated with the first release, would be still updated appropriately with the second release.
Cheers !
Pyinfra was the perfect match.
I have used their "operations" to pile up the actions that come with the next release:
- A new package (apt)
- an updtated pip dependency
- checking a file
- and docker-compose actions to update all containers
I was able define my "inventory" easily, which allows me to update my whole fleet with one single command. One caveat here, the timeout is not properly configurable with a ProxyJump. Discussion opened on stackoverflow.
Pyinfra idempotency and dry-run approaches allow me to run the update as many times as I want.
For the next release, would I need to change anything different, I will simply add a new operation. The beauty of it is that all operations are strictly executed in the order I define them, which mean that any devices that would not have been updated with the first release, would be still updated appropriately with the second release.
Cheers !
You wrote “sometimes, I have to update the docker-compose.yml on the unit, or add a library, or change something outside of the unit.”
and “I am looking for something as simple as alembic, but obviously to run scripts not DB migrations.”
I don’t see how having to run scripts follows from the first. Can’t you put the stuff that lives outside docker images in a git repo and pull changes from that?
Initial install would be a git clone and updates a git pull (maybe with a reboot to make sure nothing is running that’s using outdated stuff)
and “I am looking for something as simple as alembic, but obviously to run scripts not DB migrations.”
I don’t see how having to run scripts follows from the first. Can’t you put the stuff that lives outside docker images in a git repo and pull changes from that?
Initial install would be a git clone and updates a git pull (maybe with a reboot to make sure nothing is running that’s using outdated stuff)
The repo will provide a way to pull new scripts for new updates.
But I would still need the framework to iterate through all of them
But I would still need the framework to iterate through all of them
I think you either didn’t mention some requirement, or are too focused on “I need to run scripts to do these updates” because that’s what you use now.
To do “sometimes, I have to update the docker-compose.yml on the unit, or add a library, or change something outside of the unit.”, I don’t see what else than git pull you would need, if you put docker-compose.yml and the libraries in the repo.
rsync or other sync programs also should work.
To do “sometimes, I have to update the docker-compose.yml on the unit, or add a library, or change something outside of the unit.”, I don’t see what else than git pull you would need, if you put docker-compose.yml and the libraries in the repo.
rsync or other sync programs also should work.
Not sure I understand the problem precisely, a couple of thoughts from what I understand:
1) Aren't (Docker/OCI) containers supposed to be immutable (at least, as a best practice) ?
2) Have you looked at Pyinfra and would it help solve your problem ?
1) Aren't (Docker/OCI) containers supposed to be immutable (at least, as a best practice) ?
2) Have you looked at Pyinfra and would it help solve your problem ?
1) yes, docker containers are immutable. The pain comes more from updates that are around them
2) Trying pyinfra. Thanks for pointing it to me ! Looks promising indeed :)
Dependabot?
Thanks for the idea ! I will try pyinfra first, which provides a way to connect to the fleet and then to execute a whole set of scripts. Whereas Dependabot's perimeter looks more contained on dependencies update.
My current pain comes from the fact that, sometimes, I have to update the docker-compose.yml on the unit, or add a library, or change something outside of the unit.
I am currently writing a (python) script and a documented process for every update, which is gently becoming a pain in the xxx for the units which are not available so often, and for which I have a few updates late.
I would love to have this pile of scripts ran through automatically with a higher level command, and create the skeleton of a new migration script on demand.
Hence the question: how do you manage such remote incremental update manager ? I am looking for something as simple as alembic, but obviously to run scripts not DB migrations.