ShutIt – A Python-based shell automation framework(learnxinyminutes.com)
learnxinyminutes.com
ShutIt – A Python-based shell automation framework
https://learnxinyminutes.com/docs/shutit/
26 comments
Oh, and if you have anything you want automated, let me know - I love an automation challenge :)
Seems like another version of `Fabric`, though the expect functionality is nice. After investing so much in learning the quirks and specifics of the fab command, I'd need a very compelling reason to switch.
As someone researching whether to use Fabric or another option, I do not have to wrestle with the sunk cost fallacy. I'd be interested to hear how the features they share compare to each other objectively.
From reading through fabric they seem pretty similar. There are a few differences that leap out:
- ShutIt's pause points/interacts allow for easier debugging
- ShutIt allows for 'multisends' where you can automate more complex interaction with non-standard prompts (eg the telnet example). It's been used to automate network tasks, and complex installation tasks.
- Backgrounding tasks and managing those for faster deployment
- ShutIt has behind it a packaging system that can leverage other scripts (advanced).
I'm not a fabric expert so happy to be corrected on these.
- ShutIt's pause points/interacts allow for easier debugging
- ShutIt allows for 'multisends' where you can automate more complex interaction with non-standard prompts (eg the telnet example). It's been used to automate network tasks, and complex installation tasks.
- Backgrounding tasks and managing those for faster deployment
- ShutIt has behind it a packaging system that can leverage other scripts (advanced).
I'm not a fabric expert so happy to be corrected on these.
There are at least some cases where the expect functionality is either required, or makes things much easier.
I know expect, or expect-like functionality is used often for automating things across groups of Cisco devices, for example. Where you need to process the output of one command to drive another, or where commands have a separate "commit" Y/N type prompt.
Outside of that, most of what this describes feels like something I would do with Ansible.
I know expect, or expect-like functionality is used often for automating things across groups of Cisco devices, for example. Where you need to process the output of one command to drive another, or where commands have a separate "commit" Y/N type prompt.
Outside of that, most of what this describes feels like something I would do with Ansible.
Yeah, that was an original driver (installation of a BigIron DB (Informix) on the CL).
Sounds great for pipelining data load jobs.
Message me if you have a use case @ianmiell or on GH.
Or Ansible, which (as I understand) started merely as ssh automation.
There's something to be said for simpler tools in the same space though.
There's something to be said for simpler tools in the same space though.
Suggestion to the author: try to use portalocker [1] so that the module can also work in Windows. Currently it doesn't because the code uses the fcntl module that is *nix-only.
[1] https://pypi.python.org/pypi/portalocker
Thanks - did you try it in windows? I use CygWin at work, but it would be great to use it natively also.
Yes, it failed to import because of fcntl. Since you can't just replace fcntl with an equivalent win32api call, I suggest using portalocker.
I usually log on *nix servers from a Windows desktop, so ShutIt could be an useful addition to my toolbox - once it works on Windows, that is.
This looks like a provisioning framework rather than 'shell automation' (scripting). I couldn't work out out if it supports pipelines.
In all honesty, who is this aimed at? Some who know python and doesn't want to switch to bash, but is happy to learn the command-line tools you use in bash?
> ShutIt was built originally to facilitate the deployment of complex Docker containers, so that developers can quickly prototype builds in a structured and flexible way with as shallow a learning curve as possible.
It's for setting up developer environments? Like a sandbox? Like docker-compose?
In all honesty, who is this aimed at? Some who know python and doesn't want to switch to bash, but is happy to learn the command-line tools you use in bash?
> ShutIt was built originally to facilitate the deployment of complex Docker containers, so that developers can quickly prototype builds in a structured and flexible way with as shallow a learning curve as possible.
It's for setting up developer environments? Like a sandbox? Like docker-compose?
It's a nice API around pexpect [1] and therefor fills the same niche - spawning and interacting with console applications in a programmable way.
[1]: https://pexpect.readthedocs.io/en/stable/
[1]: https://pexpect.readthedocs.io/en/stable/
I'm familiar with expect, but I don't have any modems anymore which require AT commands to be sent to it with precise timing.
It seems like you are describing what it is, not what it should be used for (which is what I can't figure out, based on my knowledge of other tools).
It seems like you are describing what it is, not what it should be used for (which is what I can't figure out, based on my knowledge of other tools).
expect was used for a lot more than modem interactions. Certainly, there are many ways to skin this cat today, but it's not crazy to want something that removes the boilerplate of capturing output from commands and figuring out how to proceed to the next step based on what it gets.
To answer your question with specifics, you could use something like this to:
- Start services, and then do something (maybe alert, maybe make a change and try again) if it failed.
- Make configuration changes across several files and check the results of the changes (i.e. did things continue to work after the change).
- Check for updates in version control and then optionally deploy them based on the results (e.g. if there's a new tagged version).
- Interact with some API periodically, and do something if results change.
- Tons of things I don't do, but somebody else might. It's kind of an IFTTT (If This Then That) for the command line (which is maybe obvious, since I guess IFTTT was an implementation of expect for the web, in the long tradition of "take a UNIX utility and make a version for the web"). It isn't for any one thing; it's for whatever thing you want to programmatically do with stuff that maybe wasn't meant to be programmatically worked with (or maybe just doesn't have a Python API).
All of this can be done with any reasonably competent programming language, including Python (I use Perl with some tiny custom libs or bash with boilerplate, usually). This removes a bit of code. Maybe only a little code is removed, maybe a lot. It depends on what you're doing, and how well the commands or APIs you're calling lend themselves to being called from a shell or Python or whatever.
Expect was a tool to make working with tools that maybe didn't intend to be called from shell (or Tcl, or C, for that matter) programs work better when called from programs. There have always been many ways to do it. If this doesn't fit the way you think or the tools you use don't need it, there may be better ways for you to accomplish similar tasks. But, it's not a crazy idea.
To answer your question with specifics, you could use something like this to:
- Start services, and then do something (maybe alert, maybe make a change and try again) if it failed.
- Make configuration changes across several files and check the results of the changes (i.e. did things continue to work after the change).
- Check for updates in version control and then optionally deploy them based on the results (e.g. if there's a new tagged version).
- Interact with some API periodically, and do something if results change.
- Tons of things I don't do, but somebody else might. It's kind of an IFTTT (If This Then That) for the command line (which is maybe obvious, since I guess IFTTT was an implementation of expect for the web, in the long tradition of "take a UNIX utility and make a version for the web"). It isn't for any one thing; it's for whatever thing you want to programmatically do with stuff that maybe wasn't meant to be programmatically worked with (or maybe just doesn't have a Python API).
All of this can be done with any reasonably competent programming language, including Python (I use Perl with some tiny custom libs or bash with boilerplate, usually). This removes a bit of code. Maybe only a little code is removed, maybe a lot. It depends on what you're doing, and how well the commands or APIs you're calling lend themselves to being called from a shell or Python or whatever.
Expect was a tool to make working with tools that maybe didn't intend to be called from shell (or Tcl, or C, for that matter) programs work better when called from programs. There have always been many ways to do it. If this doesn't fit the way you think or the tools you use don't need it, there may be better ways for you to accomplish similar tasks. But, it's not a crazy idea.
Thanks, and well put.
Here's some things I've done (with the entire framework also - the OP is the simplified 'standalone' version):
- Developed the 'ShutItFile', eg: https://zwischenzugs.wordpress.com/2017/03/18/clustered-vm-t...
see also:
https://github.com/ianmiell/shutitfile
- Built a distro from scratch (using linux from scratch), with eerything automated: https://zwischenzugs.wordpress.com/2015/01/12/make-your-own-...
also see: https://www.youtube.com/watch?v=UGp16yvSrFE
- Put a 15-year old software stack into a container, saving enormous sums in dev costs: https://www.youtube.com/watch?v=zVUPmmUU3yY&t=16s
- Developed a 'training' tool:
https://zwischenzugs.wordpress.com/2016/04/05/linux-scales/
see also:
https://asciinema.org/a/32807?t=70
https://zwischenzugs.wordpress.com/2016/04/24/interactive-gi...
- Automated the testing of the migration of an etcd cluster: https://zwischenzugs.wordpress.com/2017/03/04/migrating-an-o...
- Win at 2048!
https://zwischenzugs.wordpress.com/2015/02/01/win-at-2048-wi...
Here's some things I've done (with the entire framework also - the OP is the simplified 'standalone' version):
- Developed the 'ShutItFile', eg: https://zwischenzugs.wordpress.com/2017/03/18/clustered-vm-t...
see also:
https://github.com/ianmiell/shutitfile
- Built a distro from scratch (using linux from scratch), with eerything automated: https://zwischenzugs.wordpress.com/2015/01/12/make-your-own-...
also see: https://www.youtube.com/watch?v=UGp16yvSrFE
- Put a 15-year old software stack into a container, saving enormous sums in dev costs: https://www.youtube.com/watch?v=zVUPmmUU3yY&t=16s
- Developed a 'training' tool:
https://zwischenzugs.wordpress.com/2016/04/05/linux-scales/
see also:
https://asciinema.org/a/32807?t=70
https://zwischenzugs.wordpress.com/2016/04/24/interactive-gi...
- Automated the testing of the migration of an etcd cluster: https://zwischenzugs.wordpress.com/2017/03/04/migrating-an-o...
- Win at 2048!
https://zwischenzugs.wordpress.com/2015/02/01/win-at-2048-wi...
I know what expect is (see my grandparent comment). I still don't know what this framework is to be used for:
> Start services, and then do something (maybe alert, maybe make a change and try again) if it failed.
It's a replacement for upstart/systemd/Launch Control?
> Make configuration changes across several files and check the results of the changes (i.e. did things continue to work after the change).
Puppet/Chef/Ansible + Sensu/Nagios?
> Check for updates in version control and then optionally deploy them based on the results (e.g. if there's a new tagged version).
cron?
> Interact with some API periodically, and do something if results change.
Also cron?
> Start services, and then do something (maybe alert, maybe make a change and try again) if it failed.
It's a replacement for upstart/systemd/Launch Control?
> Make configuration changes across several files and check the results of the changes (i.e. did things continue to work after the change).
Puppet/Chef/Ansible + Sensu/Nagios?
> Check for updates in version control and then optionally deploy them based on the results (e.g. if there's a new tagged version).
cron?
> Interact with some API periodically, and do something if results change.
Also cron?
I have several IoT devices around my home and want to push different code or configs out to them. Sure, something like Chef could work... But I'd have to install a client on each device when all I really want to do is script SSH sessions.
Great, that really helps me visualise a use case where an expect framework would shine.
Looks interesting. I'm going to give it a spin on some RPi3s I have hanging around and see how it compares to having used fabric.
One thing: the link "learnshutit.html" at the top of this page is a 404.
One thing: the link "learnshutit.html" at the top of this page is a 404.
This is incredible. The best part is that it's incredibly simple. Excellent job documenting via example.
https://github.com/ianmiell/shutit
More background here:
https://ianmiell.github.io/shutit/
This is used, for example, to automate kubernetes cluster tests for Chef scripts for OpenShift:
Chef scripts:
https://github.com/IshentRas/cookbook-openshift3
https://github.com/ianmiell/shutit-openshift-cluster
https://zwischenzugs.wordpress.com/2017/03/04/a-complete-ope...