My deployment platform is a shell script(j3s.sh)
j3s.sh
My deployment platform is a shell script
https://j3s.sh/thought/my-deployment-platform-is-a-shell-script.html
140 comments
Shell scripts are a more evolved form of programming and nobody can change my mind on that. They require less work, they're easier to make, they're flexible, compatible, composable, portable, small, interpreted, and simple. You can do more with few characters and do complex things without the complexity of types, data structures, locks, scoping, etc. You don't write complex programs in it, but you use complex programs with it, in ways that would be over complicated, buggy and time consuming in a traditional language.
That said, it's a tool. Like any tool, it depends how you use it. People who aren't trained on the tool, or don't read the instruction manual, might get injured. I'd like to see a version of it that is safer and retains its utility without getting more complicated, but it would end up less useful in many cases. Maybe that's fine; maybe it needs to be split into multiple tools.
That said, it's a tool. Like any tool, it depends how you use it. People who aren't trained on the tool, or don't read the instruction manual, might get injured. I'd like to see a version of it that is safer and retains its utility without getting more complicated, but it would end up less useful in many cases. Maybe that's fine; maybe it needs to be split into multiple tools.
I agree with most of this, my biggest issue is how hard it is for me to recall any moderately complex shell syntax (or the slightly different Makefile syntax). LLMs largely solve that for me.
The bash man page is my bible. It's dense and long but it always has the answers, you just gotta know where to look.
When people mention `bash` it's immediately a code smell. In order to write portable shell scripts, it must be only POSIX `sh`. If the need ever arises for a more complex data structure, typically I jump into AWK since it's also POSIX compliant.
Here's a note from the Ubuntu recommendation: https://wiki.ubuntu.com/DashAsBinSh
Here's a note from the Ubuntu recommendation: https://wiki.ubuntu.com/DashAsBinSh
Not every shell script has to be that portable. And even then, "POSIX compatible" shells vary in their "POSIXness". Sometimes using arrays is the right thing to do, or having a reliable indicator of the source script location, so you code to Bash 3. Sometimes associative arrays or something else is better, and your platform is known, so you code to Bash 4 or 5. Maybe all your users are Zsh users so you use that. So you can start by writing POSIX compatible scripts, but there's no sense in tying your hands if you don't need to.
The Bash manual also covers POSIX semantics, I just remember which is which. You're right that the Dash manual is a good place to check what is mostly POSIX (Dash isn't actually strictly POSIX). I would probably use Shellcheck with the correct shebang to double check what's compatible.
The Bash manual also covers POSIX semantics, I just remember which is which. You're right that the Dash manual is a good place to check what is mostly POSIX (Dash isn't actually strictly POSIX). I would probably use Shellcheck with the correct shebang to double check what's compatible.
I've never been in a situation where I had to care. Bash is everywhere, it's a de facto standard of its own. Even a lot of buildroot and Alpine based Linux deployments, which don't come with bash by default, usually have bash added to them.
It’s not everywhere though, FreeBSD for example doesn’t use it.
It doesn't come with bash pre-installed, but installing it is as simple as "pkg install bash", and that is the option that everyone working with FreeBSD will choose when they encounter a shell script that only works with bash. My point is that bash is already so ubiquitous that almost no one is going to care about an extra few megabytes for the bash executable in their OS image.
If you're installing packages anyway, then why not just write in Python, Go, or any other languages which have relatively less footguns and have more features? I'm sure at least some of those packages are also available on FreeBSD.
Eh, bash is pretty damn portable
I would RTFM so I actually learn something instead of using """AI""" regurgitation.
I find the manual for bash to be a terrible way to learn things
man pages are great if you already know 100% what you need to do, and just need a refresher.
but if you don't, and the tldr command isn't available, then good freakin luck.
but if you don't, and the tldr command isn't available, then good freakin luck.
`curl cheat.sh/$command_name` usually works pretty well! (as long as you've got curl, of course)
I have written a lot of bash. When you know what you're doing it's very productive. But it still feels like walking a tightrope, where some corner case in quoting, interpolation, comparison, etc will one day rm -rf / you.
What i really want is "python, but with really easy running of subcommands". Imagine extending python with a $ operator (prefix, applied to iterables) so that
What i really want is "python, but with really easy running of subcommands". Imagine extending python with a $ operator (prefix, applied to iterables) so that
files_iter = $('ls')
contains_pattern = $?('grep', '-q', pattern, file)
Languages like Ruby, Perl, and PHP have a backtick that can be used for that; for example in Ruby:
Also parsing ls output isn't necessarily a good idea, and what you really miss is "first class globbing" like:
>> files = `ls / | grep ^b`.split("\n")
=> ["bin", "books", "boot"]
>> p $?
#<Process::Status: pid 10889 exit 0>
>> `false`
=> ""
>> p $?
#<Process::Status: pid 10898 exit 1>
Also parsing ls output isn't necessarily a good idea, and what you really miss is "first class globbing" like:
for f in /b*; [..]
arr=(/b*)Backticks are going in the right direction, but not quite it. I want to operate on lists (or iterables etc), not strings, so i have all the usual safe and convenient facilities of the language available to build them. I want to get an object back that is extremely easy to get various kinds of results out of (Python's subprocess object isn't; i'm not familiar with Ruby's). I want more aggressive raise_for_status style error checking.
Thinking about it, this doesn't need to be an operator, and i could probably just write this myself and start trying it.
Interesting point about globbing. My feeling is that i don't actually use it a lot in scripts - the criteria for matching are usually complicated enough that i'm much more likely to use find. In a project with 1376 non-comment lines of shell script, i found eleven uses of globbing.
Thinking about it, this doesn't need to be an operator, and i could probably just write this myself and start trying it.
Interesting point about globbing. My feeling is that i don't actually use it a lot in scripts - the criteria for matching are usually complicated enough that i'm much more likely to use find. In a project with 1376 non-comment lines of shell script, i found eleven uses of globbing.
The use of ls in this way is not good form:
cd /root
for project in $(ls go-cicd);
for project in ./*
do [ -d "$project" ] || continue
...Some other nit-picks about this script.
- Everything here is done as root. For the day that you want to build with lesser privilege, do this:
- Very minor nitpick, use <<- and prefix tabs on the here document to make it slightly easier to read.
- Slight improvement, so this can print more than one argument:
- Everything here is done as root. For the day that you want to build with lesser privilege, do this:
BLDUSER=~root
- Very minor nitpick, use <<- and prefix tabs on the here document to make it slightly easier to read.
- Slight improvement, so this can print more than one argument:
println() { y=
for x
do printf %s%s "$y" "$x"
y=' '
done >> /root/gocicd.log
echo >> /root/gocicd.log # for the newline
}Why is that not good form?
It omits some characters from its output. It also mangles some others through octal escape codes or other such stuff. Depending on flags it will also not handle filenames with spaces or newlines properly. ls output is meant for human consumption, not parsing.
There are several reasons.
- Running ls forks an unnecessary process.
- The ls may be aliased with "-F" (or -p) which will corrupt the filenames.
- Environment variables may otherwise (unexpectedly) manipulate ls behavior.
- Files with spaces will not be evaluated correctly.
- Hostile files can be placed that mimic command line arguments.
The shell should evaluate filenames itself; it is very capable of doing so.
POSIX ls: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/l...
- Running ls forks an unnecessary process.
- The ls may be aliased with "-F" (or -p) which will corrupt the filenames.
- Environment variables may otherwise (unexpectedly) manipulate ls behavior.
- Files with spaces will not be evaluated correctly.
- Hostile files can be placed that mimic command line arguments.
The shell should evaluate filenames itself; it is very capable of doing so.
POSIX ls: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/l...
https://mywiki.wooledge.org/BashPitfalls#for_f_in_.24.28ls_.... describes the failure modes (and note that it's the very first pitfall because it's a common one - and one I've perpetrated plenty of times myself).
Given the situation and the project names, I wouldn't expect the use of `ls` described in TFA to ever be a problem, but doing it with a simple glob would still be nicer and is a good habit to get into overall since then you don't have to ask yourself "is this use of ls going to be safe?"
Given the situation and the project names, I wouldn't expect the use of `ls` described in TFA to ever be a problem, but doing it with a simple glob would still be nicer and is a good habit to get into overall since then you don't have to ask yourself "is this use of ls going to be safe?"
Not that I agree 100%, but the topic is covered fairly well here:
https://mywiki.wooledge.org/ParsingLs
https://mywiki.wooledge.org/ParsingLs
I was wondering if Shellcheck warned on this, and of course it does. God I love Shellcheck.
Never, ever parse or rely on the output of ls. It's very unpredictable.
You can also use webhooks to deploy with each GitHub push. The advantage over GitHub actions is you don’t have to store any secrets on GitHub or with integrators like Vercel. Just send a payload to your own endpoint each time a commit is made, and that can trigger your shell script to rebuild and deploy. Using symbolic links helps make it more robust to errors. Trigger a pull of the repo, and build. Only if the build is successful, move the symbolic link of your production app to the new build. This also allows keeping some history of builds in case you ever need to troubleshoot.
I also started with a simple shell script. Upload sources, build on the target system (golang) and restart the systemd service(es).
Then, I needed to make another machine like this, so enter Ansible. This worked well for a long time, and was relatively content with it. Along the way, I leaned about nix (and enough of it) to adopt a simple flake to pull out my tools (like golang, ansible, terraform) through. For a long time, I used it like this (e.g. still ansible, but I started building locally)
Finally, I learned enough nix to adopt NixOS. Now, I've converted my project to a nix package and a NixOS module, which allows me to totally describe the state of the machine I want. With this, remote builds and colmena (mostly for pushing secrets), I deploy a complete system, including my own software.
Then, I needed to make another machine like this, so enter Ansible. This worked well for a long time, and was relatively content with it. Along the way, I leaned about nix (and enough of it) to adopt a simple flake to pull out my tools (like golang, ansible, terraform) through. For a long time, I used it like this (e.g. still ansible, but I started building locally)
Finally, I learned enough nix to adopt NixOS. Now, I've converted my project to a nix package and a NixOS module, which allows me to totally describe the state of the machine I want. With this, remote builds and colmena (mostly for pushing secrets), I deploy a complete system, including my own software.
"i like things that work for years with as little interaction from me as possible."
Shell scripts written in NetBSD sh/Debian ash will work for as long as I live.
Shell scripts written in NetBSD sh/Debian ash will work for as long as I live.
Or, you could use NixOS and just declare your systems in some text files, git commit; git push.
You build script becomes:
I've completely removed Ansible as a result and no more python version mismatches, no more hunting endless task yaml syntax, no more "my god ansible is slow" deplyments.
You build script becomes:
while true; do
git pull
nixos-rebuild switch
sleep x
done
I've completely removed Ansible as a result and no more python version mismatches, no more hunting endless task yaml syntax, no more "my god ansible is slow" deplyments.
Instead of saying:
while true
while :
Thank you sir!
You might find it interesting to know exactly what is (and is not) in the POSIX shell. The description of the colon : operator is there.
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...
Most of the familiar userland utilities are at that website, accessible as a (somewhat crude) Apache index:
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/
Any POSIX-compliant system is required to implement the functionality described there.
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...
Most of the familiar userland utilities are at that website, accessible as a (somewhat crude) Apache index:
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/
Any POSIX-compliant system is required to implement the functionality described there.
Sounds interesting. Let's say the software is a web backend. Can you deploy it like this with zero downtime? So that the new version starts, new traffic goes to it, and the old version handles its active requests to completion and then shuts off.
I don't think so, by default I think the nixos process will simply stop (probably by sending SIGINT) the service and then start it again.
But if you could have the server into 'lame duck mode' (no new connections accepted, but existing ones can finish) / gracefull shutdown and that's a blocking call (or you could poll if it's still up etc), then you could script that before the 'nixos-rebuild switch' call. Maybe sending SIGINT to the service does that already?
But if you could have the server into 'lame duck mode' (no new connections accepted, but existing ones can finish) / gracefull shutdown and that's a blocking call (or you could poll if it's still up etc), then you could script that before the 'nixos-rebuild switch' call. Maybe sending SIGINT to the service does that already?
My current deployment method for most of my personal hosts is:
nixos-rebuild switch --target-host x.example.com
Yeah that's where I'm headed also, it's more reliable to push the configs rather than have them poll/pull automatically.
There's also https://github.com/zhaofengli/colmena which may be of interest to folks.
There's also https://github.com/zhaofengli/colmena which may be of interest to folks.
Not to deride this (too much), but the 'robustness' of deployments with shell scripts is tempting bait. Things are until they aren't, 'nobody rides for free' - decide what you're willing to pay.
Example: this interprets the output of 'ls'. Reliability is dependent on good quoting/never introducing a project with spaces
Ansible is a nice middle ground, personally. I write the state that differs, use a library of scripting.
Example: this interprets the output of 'ls'. Reliability is dependent on good quoting/never introducing a project with spaces
Ansible is a nice middle ground, personally. I write the state that differs, use a library of scripting.
Parsing ls is an anti-pattern, but the author says it works for years - we all do mistakes and as long as it works you don't notice.
And it's an easy fix:
And it's an easy fix:
- for project in $(ls go-cicd); do
+ cd go-cicd || exit 1; for project in \*; doI would suggest not to mess with the current working directory and instead do something like this:
for project in go-cicd/*; do
project="${project#*/}"Indeed, thank you for mentioning that - I was going to suggest similar.
That said, if you do (ie: scratch files, things outside of control, whatever), consider the directory stack:
https://www.gnu.org/software/bash/manual/html_node/Directory...
Using 'pushd' and 'popd' can save your fingers/brain from getting lost in context.
That said, if you do (ie: scratch files, things outside of control, whatever), consider the directory stack:
https://www.gnu.org/software/bash/manual/html_node/Directory...
Using 'pushd' and 'popd' can save your fingers/brain from getting lost in context.
I like using subshells for this:
(
cd dir
for f in ./*; [..]
)
Ah, that's a nice/neat thought - thank you for sharing! Makes total sense, temporary/disposable shell for such things
Putting
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
I knew giving specific examples would lead this way, that's not my point.
This case has been handled. Others?
There isn't anything really insightful here. Someone pleased with a script has not yet grown beyond the needs of it. Yay.
The more portable/maintainable version of this is a playbook or whatever. Someone wrote and tested a better version of whatever Work Unit as a module.
Wheel enthusiast is pleased with their finely polished, but not particularly traveled, wheel. It's great, but they're commodities and not seeing much.
I would have done the same thing with 'ansible-pull' and zero code, in even less time/effort... but I also already know Ansible and bash.
Configuration management tools are excellent at managing applications and configs. Who knew.
Something else to suggest: systemd timers. At a glance info for scheduling of the job, instead of inferred from logs that may or may not have been recorded. You can also then actually declare your deployment needs networking.
Eyeroll.jpg. This is great because the bar is so low. They'll generate a ton of useless logs if they lose networking, as-is. Long enough and the disk will fill: this is trying every minute.
This case has been handled. Others?
There isn't anything really insightful here. Someone pleased with a script has not yet grown beyond the needs of it. Yay.
The more portable/maintainable version of this is a playbook or whatever. Someone wrote and tested a better version of whatever Work Unit as a module.
Wheel enthusiast is pleased with their finely polished, but not particularly traveled, wheel. It's great, but they're commodities and not seeing much.
I would have done the same thing with 'ansible-pull' and zero code, in even less time/effort... but I also already know Ansible and bash.
Configuration management tools are excellent at managing applications and configs. Who knew.
Something else to suggest: systemd timers. At a glance info for scheduling of the job, instead of inferred from logs that may or may not have been recorded. You can also then actually declare your deployment needs networking.
Eyeroll.jpg. This is great because the bar is so low. They'll generate a ton of useless logs if they lose networking, as-is. Long enough and the disk will fill: this is trying every minute.
OK, and I wouldn't have used either Ansible or bash. I would have used picolisp, which is clearly superior for this task due to the flexibility that comes with the deep POSIX integration.
When I think about excellent software Ansible isn't the first that comes to mind either. Clearly it's different for you, and the person who wrote the TFA doesn't agree with me either.
When I think about excellent software Ansible isn't the first that comes to mind either. Clearly it's different for you, and the person who wrote the TFA doesn't agree with me either.
Let's take a step back from names for a second - I feel we're getting a little distracted by that.
My gripe isn't with the tech, or even the solution. It's perfectly fine. I know I'm being overly critical, but I think they opened themselves to some judgement by making a post!
I would do something very similar. Sure, the tool wouldn't have the exact same name, but the mechanism would be ~the same~ very similar.
If the goal is to minimize surprises, the amount of effort put into something, etc - is it not best to follow the beaten path?
They're to be commended for making a solution that works well for them, and echoing the KISS methodology, I just don't think a shell script is it. Anecdotes are funny, I guess.
It's worked well for them - but I'm here because similar things have gone terribly for me. Small decisions can have big influence
My gripe isn't with the tech, or even the solution. It's perfectly fine. I know I'm being overly critical, but I think they opened themselves to some judgement by making a post!
I would do something very similar. Sure, the tool wouldn't have the exact same name, but the mechanism would be ~the same~ very similar.
If the goal is to minimize surprises, the amount of effort put into something, etc - is it not best to follow the beaten path?
They're to be commended for making a solution that works well for them, and echoing the KISS methodology, I just don't think a shell script is it. Anecdotes are funny, I guess.
It's worked well for them - but I'm here because similar things have gone terribly for me. Small decisions can have big influence
Pretty sure bash and Perl and C (and make and autotools and probably something I'm forgetting) are the beaten path in this context, i.e. software building and deployment on Unix.
Ansible has surprised me way more often than bash has. The latter is upfront about being weird and a little bit insane, Ansible tries to have a better image. In my experience that is true also for Puppet and some other similar tools, which I'd never use without payment, at least until I can afford personal medium iron somewhere and need to provision and handle hundreds of more or less ephemeral virtual machines.
In part because I enjoy having pets rather than herds in my personal projects, but also because when using the shell or POSIX there's very little overhead, like venv and python libs and so on that will inevitably irritate me a few times a year even if I rarely use them directly.
Ansible has surprised me way more often than bash has. The latter is upfront about being weird and a little bit insane, Ansible tries to have a better image. In my experience that is true also for Puppet and some other similar tools, which I'd never use without payment, at least until I can afford personal medium iron somewhere and need to provision and handle hundreds of more or less ephemeral virtual machines.
In part because I enjoy having pets rather than herds in my personal projects, but also because when using the shell or POSIX there's very little overhead, like venv and python libs and so on that will inevitably irritate me a few times a year even if I rarely use them directly.
Bash/Perl/C being precursors don't disqualify Python/Ansible, though. Otherwise my retort would be this: assembly.
While I empathize, your experience isn't [widely] representative. A reproducible Python environment is just as attainable as any other - anyone struggling with it has accepted it, in my opinion.
Most of my peers/community uses the distribution packages. They don't even need to care about venv/pip at all.
While I empathize, your experience isn't [widely] representative. A reproducible Python environment is just as attainable as any other - anyone struggling with it has accepted it, in my opinion.
Most of my peers/community uses the distribution packages. They don't even need to care about venv/pip at all.
Or a project name that overlaps an init.d script they'd prefer to have kept
It also doesn't fail immediately if many of the commands fail (instead blindly moves on to the next statement). Consider using bash and these options: https://github.com/kvz/bash3boilerplate/blob/main/main.sh#L1... for most scripts.
It also doesn't fail immediately if many of the commands fail (instead blindly moves on to the next statement). Consider using bash and these options: https://github.com/kvz/bash3boilerplate/blob/main/main.sh#L1... for most scripts.
I assume this script runs on the server. I was building Go projects on the server as well, a vps where I have several things running. At some point I noticed that larger builds severely effected the other websites. So now I build locally and push the binary to git. To not bloat the project repo with big binary blobs I use a special deploy repo.
I think the most important part is the last lines:
"consider keeping your little things little.
it worked for little old me."
The rest are details and every one of us would implement the details in a different way.
For example a similar script could be portable with non go projects by looking for a simple build-deploy.sh script that take care of each project deployment mode/instructions.
"consider keeping your little things little.
it worked for little old me."
The rest are details and every one of us would implement the details in a different way.
For example a similar script could be portable with non go projects by looking for a simple build-deploy.sh script that take care of each project deployment mode/instructions.
As a pythonista, I am a huge fan of the plumbum library as a replacement for bash. It makes it very straightforward to run a sequence of *nix commands, but you get all the simplicity and power of the python language in terms of loops and functions and so forth. These days, I do all my server management and deployment scripts with python/plumbum.
And while simple is great, the necessary features not included in OP's scripts is that I want to spin up the new instance in parallel, verify it is running correctly, and then switch nginx or the load balancer to point to the new server. You are less prone to break production and you get zero downtime deploys.
And while simple is great, the necessary features not included in OP's scripts is that I want to spin up the new instance in parallel, verify it is running correctly, and then switch nginx or the load balancer to point to the new server. You are less prone to break production and you get zero downtime deploys.
How do you deal with plumbum not being a builtin module? Do you install it system-wide? This currently holds me back from using sh (the Python lib) for maintaining my servers, especially if I need it with root.
That's not a big deal for me since I only am running a handful of servers. I install it system-wide during initial setup of a new server. Plumbum has the ability to run remote shell commands as well, so I have a script that can login to a new remote machine and do that initial setup.
Love stuff like this. For my personal blog, I have a simple Makefile that builds the Go Binary, generates a static HTML output and then deploys it to a DigitalOcean VPS using ssh, reloads Caddy and Supervisor and boom.
Mine too, but I am not a repoman.
pushthis="rsync -avzh --del ~/path.local/ [email protected]:path.online"
pushthis="rsync -avzh --del ~/path.local/ [email protected]:path.online"
There's a lot of good in that script - it's just that it doesn't seem to cover functionalities that I'm used to after years of deploying side and "real" (business, etc.) projects to Heroku and Render.
How do you manage domain names, who deals with the ssl certificates, how do you set environment variables i.e. "secrets", how can you run postgres, how do you run remote commands i.e. dbmigrate.py, etc.
A friend and I have been working for a few months on a project to simplify this - we're not the first to do an open source IaC, but we're scratching our own itch on a lot of features that we've been missing. It's basically "deploy with git push to your own VPS and manage everything with a CLI".
I'd love to ask - what do people feel is mostly lacking from OP's script? Which features seem like the most important when deploying/managing a remote server? How do you choose if you're going to use Ansible or K8S or a script, or a full blown IaC i.e. Heroku? Is it price/ownership (i.e. having full control over the machine)/ease of use/speed of deployment/something else? Thanks!
How do you manage domain names, who deals with the ssl certificates, how do you set environment variables i.e. "secrets", how can you run postgres, how do you run remote commands i.e. dbmigrate.py, etc.
A friend and I have been working for a few months on a project to simplify this - we're not the first to do an open source IaC, but we're scratching our own itch on a lot of features that we've been missing. It's basically "deploy with git push to your own VPS and manage everything with a CLI".
I'd love to ask - what do people feel is mostly lacking from OP's script? Which features seem like the most important when deploying/managing a remote server? How do you choose if you're going to use Ansible or K8S or a script, or a full blown IaC i.e. Heroku? Is it price/ownership (i.e. having full control over the machine)/ease of use/speed of deployment/something else? Thanks!
Off topic: I love the writing style of the author and this blog. Gonna follow it.
here's the deployment script i use most often
http://canonical.org/~kragen/sw/dev3.git/hooks/post-update
it doesn't have the 60-second lag and it doesn't load the server all the time. it also doesn't run `go build` or restart a server with openrc, but those would be easy things to add if i wanted them
http://canonical.org/~kragen/sw/dev3.git/hooks/post-update
#!/bin/sh
set -e
echo -n 'updating... '
git update-server-info
echo 'done. going to dev3'
cd /home/kragen/public_html/sw/dev3
echo -n 'pulling... '
env -u GIT_DIR git pull
echo -n 'updating... '
env -u GIT_DIR git update-server-info
echo 'done.'
it doesn't have the 60-second lag and it doesn't load the server all the time. it also doesn't run `go build` or restart a server with openrc, but those would be easy things to add if i wanted them
I have a very similar system[1] for my personal projects, only I use GitHub actions to push a docker image to ECR and a commit to a config repo bumping the tag. I then have a cronjob to pull the config repo and reconcile using docker compose.
I wouldn't use it for serious stuff, but it's been working great for my random personal projects (biggest gap is if something crashes it'll stay crashed until manual intervention currently)
- [1] https://github.com/mnahkies/shoe-string-server/pull/2
I wouldn't use it for serious stuff, but it's been working great for my random personal projects (biggest gap is if something crashes it'll stay crashed until manual intervention currently)
- [1] https://github.com/mnahkies/shoe-string-server/pull/2
I use this deploy script for my hobby project: https://gist.github.com/tacone/230d5c305a9c5eff7f58ea2744f20...
It will connect over ssh, pull the code, build the containers and restart them (scripts/live is just a wrapper around docker-compose).
If the build fails, the services will keep running.
The only problem I have is that hitting CTRL+C in the very moment the containers are being restarted will leave me with the services down.
It will connect over ssh, pull the code, build the containers and restart them (scripts/live is just a wrapper around docker-compose).
If the build fails, the services will keep running.
The only problem I have is that hitting CTRL+C in the very moment the containers are being restarted will leave me with the services down.
Not exactly the same configuration as the op, but if you are developing software using Go, the combination of Caddy, a single go binary, and systemd or some other supervisor is extremely flexible and i think is the way for running multiple services on a single VM.
A shell script that deploys a couple config files and off you go. Use different accounts for each service for isolation and put all of your static files in your binary using embed.FS. No need for fancy configuration management or K8s.
A shell script that deploys a couple config files and off you go. Use different accounts for each service for isolation and put all of your static files in your binary using embed.FS. No need for fancy configuration management or K8s.
I have a similar deploy.sh script for my go projects, with a slight twist:
I compile my Go projects to a binary on a github action, scp it to the server, ssh into it and restart - all done in my deploy.sh and the GHA itself only installs Go and deps (its cached) and then calls that deploy.sh script which sits right in the repo itself.
Super happy with it. Speaking as a previous DevOps guy that got sick of AWS complexities.
I compile my Go projects to a binary on a github action, scp it to the server, ssh into it and restart - all done in my deploy.sh and the GHA itself only installs Go and deps (its cached) and then calls that deploy.sh script which sits right in the repo itself.
Super happy with it. Speaking as a previous DevOps guy that got sick of AWS complexities.
(Shameless plug) Mine is a Python script: http://piku.github.io
Why not use Ansible for something like this?
Don’t get me wrong, I love bash scripts like any other old hat, but Ansible scratches this exact itch.
You’ve got playbooks that can execute shell, provide logging, better management, history of execution, fleet management, and it’s light weight. And there’s a robust community of shared modules, etc.
Don’t get me wrong, I love bash scripts like any other old hat, but Ansible scratches this exact itch.
You’ve got playbooks that can execute shell, provide logging, better management, history of execution, fleet management, and it’s light weight. And there’s a robust community of shared modules, etc.
Why add the complexity of having to maintain an Ansible installation, a logging stack, deal with their upgrades and whatever python issue one might encounter. I had the issue of Ansible builtin `shell` not doing the right thing (sh vs bash) or it being unnecessarily slow when uselessly looking up `cowsay`.
Adding layers and layers of tooling is often overkill and it is hard to bit the simplicity of 33 lines of shell when the use case is a single person doing the code, deployment and maintenance.
Adding layers and layers of tooling is often overkill and it is hard to bit the simplicity of 33 lines of shell when the use case is a single person doing the code, deployment and maintenance.
I’m with you on the usecase. Simple server deployment on a VM, bash script is fine, in fact I recommend it. It’s when you start dealing with 5+ VMs that I would start looking into using a tool like Ansible.
Bash is better than ansible for configuring the core infrastructure underneath ansible.
In a devops workflow you "treat servers like cattle instead of pets" but your org still needs a few pets. Some host you control must either host DNS or manage your DNS provider's API key. Same for CA, IdP, git, backup and monitoring services, and the ansible machine itself. You'll have to manually configure these things before your "cattle" tools can run.
Once you're up and running, it's possible to make ansible manage it own dependencies, but this introduces circular dependencies complicates bootstrapping (consider a disaster recovery situation) and amplifies both the impact of faults and the difficulty of troubleshooting them. Do you want to be debugging python dependencies in the middle of the night so you can finally get ansible to execute the couple bash commands that will bring your ACME CA back up? I'd rather run bash directly.
At a small scale with a stable set of requirements, your core infrastructure is better served by a good operations manual and a simple deployment toolset with minimal dependencies. Plain bash fits the bill!
In a devops workflow you "treat servers like cattle instead of pets" but your org still needs a few pets. Some host you control must either host DNS or manage your DNS provider's API key. Same for CA, IdP, git, backup and monitoring services, and the ansible machine itself. You'll have to manually configure these things before your "cattle" tools can run.
Once you're up and running, it's possible to make ansible manage it own dependencies, but this introduces circular dependencies complicates bootstrapping (consider a disaster recovery situation) and amplifies both the impact of faults and the difficulty of troubleshooting them. Do you want to be debugging python dependencies in the middle of the night so you can finally get ansible to execute the couple bash commands that will bring your ACME CA back up? I'd rather run bash directly.
At a small scale with a stable set of requirements, your core infrastructure is better served by a good operations manual and a simple deployment toolset with minimal dependencies. Plain bash fits the bill!
I think even ansible is overkill for such a simple thing. Ansible use case works better when you need to do stuff on multiple hosts.
For years I've started using and abandoned ansible and puppet recipes for setting up my own computers and everytime the conclusion was that I would spend more time installing git, ansible and puppet in the first place and debugging my recipes than using them. Now all my setup lives in shell functions in my .bashrc.d. I still need git but I don't need ansible or puppet anymore.
For years I've started using and abandoned ansible and puppet recipes for setting up my own computers and everytime the conclusion was that I would spend more time installing git, ansible and puppet in the first place and debugging my recipes than using them. Now all my setup lives in shell functions in my .bashrc.d. I still need git but I don't need ansible or puppet anymore.
Ansible is great even for simple single-host 'shell scripts'.
Lean into the module ecosystem. Want to ensure a config file is a certain way? Jinja/template it, or use lineinfile instead of echo/shell redirects.
That's a lot of mumbo-jumbo. The point is, there's a lot of stuff scripts want to do. Ansible provides these as modules. Using the modules spares you from writing code to do something in a robust/repeatable way.
The 'line in a file' example is a good case, IMO. A shell script with redirection either requires specific code to look first, or simply endlessly append. With Ansible you don't have to do all of that.
Your script needs to do something when something changed? Ansible has you covered: handlers!
Python is right within reach too. I find it a way to write Python via YML, basically.
Lean into the module ecosystem. Want to ensure a config file is a certain way? Jinja/template it, or use lineinfile instead of echo/shell redirects.
That's a lot of mumbo-jumbo. The point is, there's a lot of stuff scripts want to do. Ansible provides these as modules. Using the modules spares you from writing code to do something in a robust/repeatable way.
The 'line in a file' example is a good case, IMO. A shell script with redirection either requires specific code to look first, or simply endlessly append. With Ansible you don't have to do all of that.
Your script needs to do something when something changed? Ansible has you covered: handlers!
Python is right within reach too. I find it a way to write Python via YML, basically.
> Using the modules spares you from writing code to do something in a robust/repeatable way.
That is a huge lie: Declarative code is still code. Using modules is similar to reusing functions. The things is, while reusability and declarative code is nice when you want to deploy and manage multiple machines and have an automated network install that bootstrap your automated configuration tool. It is worth the effort because there are many machines but all that automation need to be tested/fixed on a regular basis (distro releases, etc). If you are reinstalling your machine from an usb pendrive, or image once every so many full moons, you need first to bootstrap ansible and the playbook. How do you that in an idempotent manner? The time you have done that you are probably already ready.
The only thing I need on my dev machines is :
- my software configs: comes with a git repo of my dotfiles.
- a dev directory: mkdir is idempotent, it will not destroy dir if it exists so no need for a declarative language
- some packages: a single package install is needed. While using a configuration tool allows you to declare package name depending on distro version and regardless of package tool, usually someone who is managing 2-3 machines stick to one OS so in my case a single `dnf install -y <list of package>` is enough.
- a few tools I curl from github or other places. I have one bash function for each of them to get latest release (a one liner) and one to compare installed version with latest release and install if needed. Ansible doesn't do a better job at it. I checked ansible-galaxy for some of the tools I download, for some no module exists, for those that have modules, they are just made of ... shell scripts called by an ansible task that is larger than my own script. See example[1]
- a few desktop files, they come with my dotfiles git repo, no need to "declare them"
- a handful of stuff that comes with an install shell script (the infamous curl | bash). Ansible don't help me much or I'd have to rewrite the install script as an ansible playbook and maintain it myself forever.
No handler necessary, none of this requires a reboot or a service restart.
Also Ansible is probably the worst example as it is an half assed declarative language that doesn't even encourage idempotence. Basically it is made by and for old unix guys who want to continue writing sequencial scripts the old and crappy way while pretending they do things in a modern way. And it is the reason it has become so popular against cfengine, puppet, chef and salt. I don't see the point of using ansible if it is to have the same low standard of quality as plain old scripts.
My experience working with teams using ansible has only reinforced my view that this language is for people who like to do things the dirtiest way anyway but wrapping it in a declarative language so they can put devops engineer in their resume.
[1] https://github.com/andrewrothstein/ansible-eksctl/tree/main
That is a huge lie: Declarative code is still code. Using modules is similar to reusing functions. The things is, while reusability and declarative code is nice when you want to deploy and manage multiple machines and have an automated network install that bootstrap your automated configuration tool. It is worth the effort because there are many machines but all that automation need to be tested/fixed on a regular basis (distro releases, etc). If you are reinstalling your machine from an usb pendrive, or image once every so many full moons, you need first to bootstrap ansible and the playbook. How do you that in an idempotent manner? The time you have done that you are probably already ready.
The only thing I need on my dev machines is :
- my software configs: comes with a git repo of my dotfiles.
- a dev directory: mkdir is idempotent, it will not destroy dir if it exists so no need for a declarative language
- some packages: a single package install is needed. While using a configuration tool allows you to declare package name depending on distro version and regardless of package tool, usually someone who is managing 2-3 machines stick to one OS so in my case a single `dnf install -y <list of package>` is enough.
- a few tools I curl from github or other places. I have one bash function for each of them to get latest release (a one liner) and one to compare installed version with latest release and install if needed. Ansible doesn't do a better job at it. I checked ansible-galaxy for some of the tools I download, for some no module exists, for those that have modules, they are just made of ... shell scripts called by an ansible task that is larger than my own script. See example[1]
- a few desktop files, they come with my dotfiles git repo, no need to "declare them"
- a handful of stuff that comes with an install shell script (the infamous curl | bash). Ansible don't help me much or I'd have to rewrite the install script as an ansible playbook and maintain it myself forever.
No handler necessary, none of this requires a reboot or a service restart.
Also Ansible is probably the worst example as it is an half assed declarative language that doesn't even encourage idempotence. Basically it is made by and for old unix guys who want to continue writing sequencial scripts the old and crappy way while pretending they do things in a modern way. And it is the reason it has become so popular against cfengine, puppet, chef and salt. I don't see the point of using ansible if it is to have the same low standard of quality as plain old scripts.
My experience working with teams using ansible has only reinforced my view that this language is for people who like to do things the dirtiest way anyway but wrapping it in a declarative language so they can put devops engineer in their resume.
[1] https://github.com/andrewrothstein/ansible-eksctl/tree/main
> reinstalling your machine from an usb pendrive, or image once every so many full moons, you need first to bootstrap ansible and the playbook. How do you that in an idempotent manner?
You just reinstalled; do you really care if the preparation is idempotent?
Anyway: kickstart is how I deal with that. The way one automates installations. Anyone reinstalling their workstation that often should probably look into it.
It wants a list of packages that get installed by default, Ansible is one of them. The install environment makes Ansible available, then runs ansible-pull to fetch the repository and run the play.
I hear you now: "but USB installs!"
Who is this person that does this so often to automate it, but accept clicking through the UI/installer and so on? Set up tftp and PXE already, you're neck deep.
The unix greybeards would put shell scripts in those kickstarts. I feel it's slightly improved by using playbooks held externally in SCM.
The module library doesn't cover everything, but it's great for routine system administration. It may not have the latest whizbang API.
Ansible is useful, I'm not debating this. One can write it as poorly as you represent, but they don't have to.
I do lament people writing it like scripts. They miss the point, we're in agreement there. The core modules are idempotent if used well.
The Command/Script modules shouldn't even exist in my opinion. Force people to custom-fact those things. It can be a plain text file or a robust script.
You just reinstalled; do you really care if the preparation is idempotent?
Anyway: kickstart is how I deal with that. The way one automates installations. Anyone reinstalling their workstation that often should probably look into it.
It wants a list of packages that get installed by default, Ansible is one of them. The install environment makes Ansible available, then runs ansible-pull to fetch the repository and run the play.
I hear you now: "but USB installs!"
Who is this person that does this so often to automate it, but accept clicking through the UI/installer and so on? Set up tftp and PXE already, you're neck deep.
The unix greybeards would put shell scripts in those kickstarts. I feel it's slightly improved by using playbooks held externally in SCM.
The module library doesn't cover everything, but it's great for routine system administration. It may not have the latest whizbang API.
Ansible is useful, I'm not debating this. One can write it as poorly as you represent, but they don't have to.
I do lament people writing it like scripts. They miss the point, we're in agreement there. The core modules are idempotent if used well.
The Command/Script modules shouldn't even exist in my opinion. Force people to custom-fact those things. It can be a plain text file or a robust script.
That is exactly what I am saying, nobody does it that often, so automatising is not even an option usually as things move so fast a playbook would have to be debugged and modified every time you use it.
I think Ansible is a little overkill for some projects tbh. Ideally I'd love a middle ground between bash scripts and Ansible, similar to Caddy's config simplicity over nginx.
>it’s light weight
Eh, don't think that's the case for everyone.
I dabbled with Ansible at a previous job, and set up a very basic personal server setup for Nextcloud and one other app. It was much slower than if I had just written some bash scripts. Idempotency was nice, but the feedback loop wasn't great.
>it’s light weight
Eh, don't think that's the case for everyone.
I dabbled with Ansible at a previous job, and set up a very basic personal server setup for Nextcloud and one other app. It was much slower than if I had just written some bash scripts. Idempotency was nice, but the feedback loop wasn't great.
Related: https://github.com/containrrr/watchtower
Polls a docker registry and automatically restarts the container with the same flags that it was started with using the latest image.
Polls a docker registry and automatically restarts the container with the same flags that it was started with using the latest image.
Scripts get complicated when people start worrying about "portability," like we were in 1991 so we have to use "sh".
It's 2024 and you're not developing the next vim or Postgres. Use bash.
It's 2024 and you're not developing the next vim or Postgres. Use bash.
For PHP https://deployer.org
For JS https://webpod.dev
Laravel has Envoy as well: https://laravel.com/docs/11.x/envoy
I've played around a bit with Deployer for some projects. It's decent, but feels brittle. It's very dependent on you sticking with its assumed default setup, docs are all over the place, and extending/replacing scripts I found confusing.
I moved back to bash scripts.
I've played around a bit with Deployer for some projects. It's decent, but feels brittle. It's very dependent on you sticking with its assumed default setup, docs are all over the place, and extending/replacing scripts I found confusing.
I moved back to bash scripts.
I agree that sometimes Bash is enough, which is what I show in Deployment from Scratch. However I am moving pretty much everything to Kamal now...
Guessing you're talking about https://kamal-deploy.org/ which looks interesting, though I tend to like reconciliation logic based systems ... but often only fired off imperatively with a plan/apply separation. So I shall be having a poke around anyway :)
Got your Kamal book. It was much needed, great resource. Thanks for that!
If the author is reading this: did you code the fish animation CSS manually or used some wrapper for all these moz- and webkit- variants?
It’s a fish? I thought it was a pocket watch
Well, that would make it quite an exotic watch: https://j3s.sh/static/unnamed-puffy.png
The heredoc in the script is never terminated. Probably it's not the production version but one for publishing ;-)
Can someone ELI5 what he said in the blog post please? What does the script do?
The script does a git fetch
If the current code is behind (there are new commits), it merges them. If it fails, it stops.
If not, it runs `go build`. If it fails, it stops.
If not, it moves the binary to the right location and restarts the service.
If the current code is behind (there are new commits), it merges them. If it fails, it stops.
If not, it runs `go build`. If it fails, it stops.
If not, it moves the binary to the right location and restarts the service.
I can't speak to the validity for the author's use case as I'm not a golang dev, but in spirit I do like the idea. I think this trend back to simplicity (monoliths, sqlite, bash scripts) makes it good timing to be posting and learning things like this.
Especially as more and more new, easy/low config tools come out like Caddy, this gets simpler over time.
I have a testbed boilerplate project for Laravel in which server provisioning & deployments are done by bash scripts over SSH. Excluding comments/spacing, the provisioning script is 35 lines of mostly installing dependencies and minor file template copies. For simpler projects not needing queue workers and/or not using more "exotic" tech like Laravel Octane, this could probably be cut down to 30.
TL;DR do the simplest thing that works for you and move on with life - the value in your project, if you intend to deploy it, is for it to be used.
Especially as more and more new, easy/low config tools come out like Caddy, this gets simpler over time.
I have a testbed boilerplate project for Laravel in which server provisioning & deployments are done by bash scripts over SSH. Excluding comments/spacing, the provisioning script is 35 lines of mostly installing dependencies and minor file template copies. For simpler projects not needing queue workers and/or not using more "exotic" tech like Laravel Octane, this could probably be cut down to 30.
TL;DR do the simplest thing that works for you and move on with life - the value in your project, if you intend to deploy it, is for it to be used.
I’d like to think that there is in fact a trend towards simplicity. But with AI and AI written code, I unfortunately think we may be heading to having even more opaque code.
I always say: my pipeline is a shell script I call pipeline.sh
> my script will never:
> - go down
I've had cron log files get too big and causes issues.
> - require an upgrade
I can't count the number of times unattended-upgrades has broken something.
> - force me to migrate
Let's hope the OS is still receiving security updates, because installing on a VPS like this always has a high migration cost.
This sort of deployment is a fair starting point, but let's not pretend it's some perfect ideal.
....
Look. For deploying a blog, sure, but no one is deploying their blog on k8s. There is a reason why big complex deployment and orchestration systems exist, because there are use-cases for them. This is not one of them, but there's no need to stick your head in the sand over requirements and pretend they don't exist.
> - go down
I've had cron log files get too big and causes issues.
> - require an upgrade
I can't count the number of times unattended-upgrades has broken something.
> - force me to migrate
Let's hope the OS is still receiving security updates, because installing on a VPS like this always has a high migration cost.
This sort of deployment is a fair starting point, but let's not pretend it's some perfect ideal.
....
Look. For deploying a blog, sure, but no one is deploying their blog on k8s. There is a reason why big complex deployment and orchestration systems exist, because there are use-cases for them. This is not one of them, but there's no need to stick your head in the sand over requirements and pretend they don't exist.
Limitations are a good thing
my script will never:
- go down
- require an upgrade
- force me to migrate
- surprise me
- keep me up at night
but my server will:
- require direct source code access
- require go build tools
- require maintaining GitHub auth
- require upgrading build tools over time
- not be trivial to rebuild on failureechelon_musk(2)
Yes more you don’t need
I sadly don't recommend it for my day job, simply because of liability. When something messes up with ansible, terraform, docker, cloudformation etc, no-one gets any blame because 'complex systems', 'it happens' etc etc; with a simple script going wrong, they would hang me high even though it probably saves a crap load in maintenance, compute etc over the 10s of years. Same reason we use clusters and IaC while of course nothing we do needs it; if an aws cluster goes down, no-one but aws gets blamed, while if the $2 postgres vps@cheapafhosting (with an higher uptime than that aws cluster by the way; human error downed it a few times, short, but still) is down even for a ping, everyone is upset and pointing fingers.