Times I've Messed Up as a Developer(medium.com)
medium.com
Times I've Messed Up as a Developer
https://medium.com/@zacharykuhn/the-times-ive-messed-up-as-a-developer-3c0bcaa1afd6
136 comments
This is why I do a "mv" to "tmp" instead of using "rm" when I can. Having special messages or colours to differentiate between production and other environments is a great idea as well. Generally if I'm about to do something on production, I'll close everything else to do with other environments as it's just too easy to slip up.
There's trash-cli that moves files to "trash". Alias "rm" to display an error message to avoid accidental deletions.
https://github.com/sindresorhus/trash-cli
https://github.com/sindresorhus/trash-cli
You can also alias "rm -i" or "rm -I" to "rm" to avert major catastrophes.
From man rm:
-i prompt before every removal
-I prompt once before removing more than three files, or when removing recursively; less intrusive than -i, while still giving protection against most mistakes
From man rm:
-i prompt before every removal
-I prompt once before removing more than three files, or when removing recursively; less intrusive than -i, while still giving protection against most mistakes
This alias seems to be set by default in many linux distributions these days (and has been that way for many years now).
The problem is that you get used to this crutch and rely on it, until one day you're logged in someplace without the alias and end up deleting something you shouldn't.
I'm not a fan and normally disable it. Instead, I prefer to be very careful and deliberate when using rm, especially with any wildcards, making sure I'm in the right directory and that the pattern matches what I expect it to.
The problem is that you get used to this crutch and rely on it, until one day you're logged in someplace without the alias and end up deleting something you shouldn't.
I'm not a fan and normally disable it. Instead, I prefer to be very careful and deliberate when using rm, especially with any wildcards, making sure I'm in the right directory and that the pattern matches what I expect it to.
I'll probably get shit on for admitting it, but I generally keep rm aliased to `rm -rf`. Recursive, force.
It forces me to be careful and deliberate. There's never a safe option or a backstop. I treat rm like a gun - always assume it's loaded and going to murder whatever I point it at without hesitation. If I'm not 100% sure about what I'm about to delete or how a wildcard will expand, then I echo it first.
I messed up once, about 15 years ago now. Lost a few personal files. Haven't screwed up since.
It forces me to be careful and deliberate. There's never a safe option or a backstop. I treat rm like a gun - always assume it's loaded and going to murder whatever I point it at without hesitation. If I'm not 100% sure about what I'm about to delete or how a wildcard will expand, then I echo it first.
I messed up once, about 15 years ago now. Lost a few personal files. Haven't screwed up since.
> This alias seems to be set by default in many linux distributions these days (and has been that way for many years now)
Which Linux distributions do this? I have not seen this alias set by default on any distribution that I've used recently (Gentoo, Linux Mint, Debian, Ubuntu, Crunchbang, Arch).
Which Linux distributions do this? I have not seen this alias set by default on any distribution that I've used recently (Gentoo, Linux Mint, Debian, Ubuntu, Crunchbang, Arch).
I stand corrected, it doesn't seem to be the case anymore.
I can find old rpms and deb files with rm aliased in /etc/skel/.bashrc or /etc/skel/profile/aliases.sh, but the most recent I found is in a bash 3.2 package, so this practice seems to have stopped quite a few years ago.
I can find old rpms and deb files with rm aliased in /etc/skel/.bashrc or /etc/skel/profile/aliases.sh, but the most recent I found is in a bash 3.2 package, so this practice seems to have stopped quite a few years ago.
An then use 'yes' to avoid all those key pressing :)
my go-to .bashrc hashes the hostname into a color and prints the prompt in that color. it has saved me from making a dumb mistake a few times.
Production prompts are red.
Staging prompts are yellow.
Development prompts are green.
Staging prompts are yellow.
Development prompts are green.
yeah, that'd be great. but our staging units (i don't deal with production units) come and go so often that keeping a list of hostnames became unrealistic.
Is there something in your dev process that would actually require you to implement it with a list of hostnames, rather than just putting the "staging" version of .bashrc onto the staging machines, and the "dev" version onto dev machines?
Mine is exactly the same :-)
Could you post the code somewhere?
sure, although my technique is a bit kuldgy.
https://gist.github.com/woodrowbarlow/52b02ab27d0f8088891c3d...
look at `get_hash_color`.
https://gist.github.com/woodrowbarlow/52b02ab27d0f8088891c3d...
look at `get_hash_color`.
Two terms was such a common source of mistakes: I shut down a server parked in a nearby country and we had to drive there to turn it on.
Since then I’ve adopted custom prompt and color scheme for production.
I really wish there was some practical information course at compsci with these kind of tips to prevent the most common slippages
Since then I’ve adopted custom prompt and color scheme for production.
I really wish there was some practical information course at compsci with these kind of tips to prevent the most common slippages
> I really wish there was some practical information course at compsci with these kind of tips to prevent the most common slippages
This is everyday Operations work. There is no school that teaches the eclectic mix of OS and container, networks and routes, app/db/code, CI/CD, BC/DR, writing KB articles, running standups and meetings, triage technical issues, configuration management, Change Control, and Incident Management.
These diverse skill sets are often called "Devops", but nobody makes a Devops degree. The only way to assemble this experience is while working to support developers.
This is everyday Operations work. There is no school that teaches the eclectic mix of OS and container, networks and routes, app/db/code, CI/CD, BC/DR, writing KB articles, running standups and meetings, triage technical issues, configuration management, Change Control, and Incident Management.
These diverse skill sets are often called "Devops", but nobody makes a Devops degree. The only way to assemble this experience is while working to support developers.
I also wonder how effective a program can be at teaching this kind of thing. No amount of studying ever ingrained something in my mind as well as the adrenaline shot of somehow fucking up production. I'm not aware of any pedagogical methods to engage the endocrine system, but I find it very useful for retention.
Well, part of the learning is making the mistakes yourself, and part of it is your paranoid coworkers cautioning you against certain behavior and relaying the horror stories of what it might result in as experienced by themselves or someone they knew. A thedailywtf.com for operations mishaps would probably go a long way.
There's a theory that the same person should not be both a Developer and an Operator for the same project - that crucial mental separation between Making something and Keeping It Online gets too blurred when one person wears both hats. I've witnessed this in my own personal and professional projects. I guess it is a counter-argument to the DevOps/Agile philosophy.
> Since then I’ve adopted custom prompt and color scheme for production.
An AHA moment for me. I've always wondered why people cared about all the terminal customization options (I grew up typing on an 80x48 AAA glass TTY). This is a good reason.
I've used a custom prompt with the host name built in for decades, but this kind of color issue makes sense!
An AHA moment for me. I've always wondered why people cared about all the terminal customization options (I grew up typing on an 80x48 AAA glass TTY). This is a good reason.
I've used a custom prompt with the host name built in for decades, but this kind of color issue makes sense!
I always have so many terminals open that this is a must for me. I have different colors for production and I used ansible to make them consistent. It's nice to not have to waste brain power trying to guess which terminal I'm on.
One the last day of a colleague of mine, he decided to run `rm -rf --no-preserve-root /` to see what actually would happen on his workstation (it was going to be formatted shortly after). We were all standing around talking waiting for whatever would happen to happen before we saw him off. A few minutes in I asked if he had unmounted our NFS volumes. Nope. He quickly pulled the power, and that was that. I have no idea if anything was deleted, but he probably had ~ 2PB of mounted data attached to his workstation. Everything from build archives, customer logs dumps, test artifacts, dogfooding clusters (the product was a parallel distributed networked file system), and possibly a few random other things.
No idea if anything would have happened, most of the data was read only (builds, logs, etc) but some was not.
No idea if anything would have happened, most of the data was read only (builds, logs, etc) but some was not.
Oh fuck moment.
This is why --one-file-system should be default, with a warning if another file system is encountered that's skipped so if it's what is desired you can re-run it (or umount and try again).
I used to setup my shell so that the text on the production servers was red and local it was white. Made it easy to know which machine I was tying into :)
> I used to setup my shell so that the text on the production servers was red
I do the same. People always walk up and make some snarky comment like "wow, nice background color" and then I explain why I have a red terminal background and they say "oh... that's a great idea"
I do the same. People always walk up and make some snarky comment like "wow, nice background color" and then I explain why I have a red terminal background and they say "oh... that's a great idea"
I tried this for a while after a goof up. However because my dev environments more or less looked like default, if I forgot to setup a new production server to change the color, I'd mistake it for local at some point.
So I change my dev to be the non-default now. If I see the default/typical color scheme, I assume production.
So I change my dev to be the non-default now. If I see the default/typical color scheme, I assume production.
That’s a pretty good idea. “Be careful” is a good default.
To prevent this, my iTerm2 has a custom background enabled when logged in on any production servers.
I wish the same were easily possible on a Linux terminal without much "trickery" :|
I have a different 'theme' for tmux on any production servers I set up than development/test servers and my local machine. It ends up that I nest my remote sessions in my local, but it's effective.
Whats wrong with just setting PS1 to indicate that its a production server, and using a decent host naming scheme that properly indicates the function of the machine you're logged into? I mean this is what I do for my Linux machines, and I haven't deleted a Linux production machine in decades ..
Maybe this is an obvious question but...how do you do that if you ssh into your remote machines through a bastion that normalizes everyone to the same end user? I'm unable to set a custom .bashrc on the remote server, because its a shared user, so how would you change the ps1 on the remote box?
> Moral of the story - Don't let anyone give you access to production servers.
If you are root, make another user. you can delete it later.
If you are root, make another user. you can delete it later.
> Moral of the story - Don't let anyone give you access to production servers.
In my case, it was "Don't let anyone have you do testing/debugging on the production machine."
I was 18, first dev job, small mom-n-pop which developed insurance claims management software. Early 1990s, green screen VT100 terminals connected remotely via modems and dedicated phone lines.
Anyhow, something was wrong, I forget what, and to fix it I needed to debug using a real dataset. Between each run I had to clear out and rebuild a series of test "tables" (that's in quotes because the system was PICK, and the DB isn't exactly a SQL RDBMS), then copy the production tables to the test tables, and re-run.
Well - at one point I cleared the wrong table - cleared the production and not the test table.
Within seconds all the phones lit up in the office. I groaned (rather loudly), because I knew I just dun f'd up. The client was one of 3 or 4 that the company had, and the largest. Needless to say, for every minute they were down, they couldn't process claims, and were losing money.
Did I mention there weren't any backups? Not that backups weren't made, just that they were "end-of-day" backups, so the backup for that day's work (head's down manual entry of claims for processing) had not yet happened.
Fortunately for me, the table I deleted could be recreated (mostly) from two other still-intact tables. My manager and I worked thru the evening rebuilding the table as much as we could, and also pulled in data from the prior day's backup to make it as complete as possible. Even so, they still had to re-enter some data lost that wasn't recoverable by that method.
Did we set up a test server or do anything else that any sane company would do after that incident? Nope. Things just continued on in the same manner, and somehow I kept my job. Honestly, we should have been doing the testing in-house, but this was using an IBM RS6K box that at the client was much larger than what we had as a dev box (we were using the smallest version of that machine at the time, which was meant to be a single user desktop workstation as I later found out), so we couldn't fit the data onto the hard drive, even if it was possible to download it all via modem (it wasn't - I'd probably still be waiting for the dump to complete given the modem was only 9.6 kbps). I have no idea what it would have cost to do it right, but the drives needed back then, coupled with the infamous IBM service contract - meant we couldn't just hop over to the nearest Insight to grab a cheap hard drive and slap it in.
Today, I'd probably have the admin spin up another instance of the PICK system as another process on AIX, and (somehow) replicate the live data/code over and run in that instance. Still not "safe", but probably would have been much safer than what we were doing. But alas, I was but a kid at the time, and barely understood *nix (but that was where I first learned about it - and I also gained a healthy lesson/understanding/fear/respect for terminfo - for the purposes of supporting a variety of serial terminals and PC terminal emulators).
In my case, it was "Don't let anyone have you do testing/debugging on the production machine."
I was 18, first dev job, small mom-n-pop which developed insurance claims management software. Early 1990s, green screen VT100 terminals connected remotely via modems and dedicated phone lines.
Anyhow, something was wrong, I forget what, and to fix it I needed to debug using a real dataset. Between each run I had to clear out and rebuild a series of test "tables" (that's in quotes because the system was PICK, and the DB isn't exactly a SQL RDBMS), then copy the production tables to the test tables, and re-run.
Well - at one point I cleared the wrong table - cleared the production and not the test table.
Within seconds all the phones lit up in the office. I groaned (rather loudly), because I knew I just dun f'd up. The client was one of 3 or 4 that the company had, and the largest. Needless to say, for every minute they were down, they couldn't process claims, and were losing money.
Did I mention there weren't any backups? Not that backups weren't made, just that they were "end-of-day" backups, so the backup for that day's work (head's down manual entry of claims for processing) had not yet happened.
Fortunately for me, the table I deleted could be recreated (mostly) from two other still-intact tables. My manager and I worked thru the evening rebuilding the table as much as we could, and also pulled in data from the prior day's backup to make it as complete as possible. Even so, they still had to re-enter some data lost that wasn't recoverable by that method.
Did we set up a test server or do anything else that any sane company would do after that incident? Nope. Things just continued on in the same manner, and somehow I kept my job. Honestly, we should have been doing the testing in-house, but this was using an IBM RS6K box that at the client was much larger than what we had as a dev box (we were using the smallest version of that machine at the time, which was meant to be a single user desktop workstation as I later found out), so we couldn't fit the data onto the hard drive, even if it was possible to download it all via modem (it wasn't - I'd probably still be waiting for the dump to complete given the modem was only 9.6 kbps). I have no idea what it would have cost to do it right, but the drives needed back then, coupled with the infamous IBM service contract - meant we couldn't just hop over to the nearest Insight to grab a cheap hard drive and slap it in.
Today, I'd probably have the admin spin up another instance of the PICK system as another process on AIX, and (somehow) replicate the live data/code over and run in that instance. Still not "safe", but probably would have been much safer than what we were doing. But alas, I was but a kid at the time, and barely understood *nix (but that was where I first learned about it - and I also gained a healthy lesson/understanding/fear/respect for terminfo - for the purposes of supporting a variety of serial terminals and PC terminal emulators).
Did anyone ever notice that it happened?
No one. It was lunchtime and it's possible no ne traded so far that day.
Well played
The flip side to the story..
what if he messed up the restore, and instead of restoring from the only known backup, he wiped our the only known backup?
Would be epic fired story.
I vote for admit when you mess up, and ask for help. Don't be too proud, we all mess up.
what if he messed up the restore, and instead of restoring from the only known backup, he wiped our the only known backup?
Would be epic fired story.
I vote for admit when you mess up, and ask for help. Don't be too proud, we all mess up.
I wonder if kids these days manage to get as much experience recovering a borked computer? I vaguely remember poking at autoexec.bat and config.sys files to get games working, or to attempt to revert some screw-up when I failed to get one working. Or becoming very good at using Linux live CDs to grab personal docs/tax information/business slips... off a hard-disk before wiping it and reinstalling Windows, after a bad driver update or dodgy install of some software.
The first few times, you get that icy "oh shit" feeling, but figuring out how to fix it, and that it can (almost) always be fixed, is invaluable.
The first few times, you get that icy "oh shit" feeling, but figuring out how to fix it, and that it can (almost) always be fixed, is invaluable.
Well, I know I how to, and I am fairly young.
I've seen a DBA drop a production table as well, fortunately it was a small in memory table for non persisting data that could be quickly repopulated. Still lead to about 5 mins of downtime.
I always have two terminals open, both running tmux. One is white text on black for dev work, the other is white text on red. I only ever ssh into production servers on the white on red terminal so there's a strong visual signal that I absolutely shouldn't be running drop, delete, truncate, rm, dd, whatever commands on them and if I do then I am extremely careful.
I always have two terminals open, both running tmux. One is white text on black for dev work, the other is white text on red. I only ever ssh into production servers on the white on red terminal so there's a strong visual signal that I absolutely shouldn't be running drop, delete, truncate, rm, dd, whatever commands on them and if I do then I am extremely careful.
> I only ever ssh into production servers on the white on red terminal so there's a strong visual signal
This is a great idea.
This is a great idea.
I've found that quickly changing the color of Terminal windows is pretty much the only useful application for the MacBook Pro Touch Bar.
You have to back it into your profile. That way ANYONE that SSHes into that box gets the funny text colors... or at least a funny colored prompt.
Good way to save your bacon.
Good way to save your bacon.
"One is white text on black for dev work, the other is white text on red."
Cool. Sorry, but I'm going to steal this idea from now on.
Cool. Sorry, but I'm going to steal this idea from now on.
Also doable with desktops for those occasions where it is a remote screen share.
For UAT / Test environments that are different than dev, I use a yellow background.
For UAT / Test environments that are different than dev, I use a yellow background.
I used to do the same thing with RDPs to windows production servers. I'd set up a different colored color scheme on each computer, with prod being bright red :)
For Windows work, I made sure to have one regular user and one admin user. And set the desktop background of the admin user to red. An additional benefit is that the garish background is uncomfortable to work with, and so discourages being logged in as admin all the time...
Changing the desktop background alone isn't enough. You will inevitably get into a situation where fullscreen windows are occluding it. To avoid any ambiguity, set the color scheme to high contrast and then set all the windows title bars to red, and tint the task bar red. On QA, I tint everything purple, and on DR I tint everything green, or yellow.
Thanks for sharing , I setup now a production profile for Konsole with red on black for the tab I use for ssh into servers
I once got a call from someone with DBA privileges: "I just ran 'SELECT table_name FROM user_tables' and it says 'no rows selected." I logged in and checked, and said, 'Yep, you have no tables.' Fortunately, Oracle by then had the RECYCLE$BIN, and the interruption in service was minimal. I'm not sure how he dropped all the tables.
And let's not talk about my own (mostly minor) disasters.
And let's not talk about my own (mostly minor) disasters.
Not quite so disastrous, but still pretty dumb.
I was logged in over VPN to a client machine. I suspected the problem I was fixing was something to do with the network misbehaving. It was a Windows machine.
I thought "I know: I'll disable and reenable the network device!".
A few moments later: "Oh. Bugger.".
Shortly followed by an embarrassing call to the client's IT department to ask them to reenable the network device on that machine...
I was logged in over VPN to a client machine. I suspected the problem I was fixing was something to do with the network misbehaving. It was a Windows machine.
I thought "I know: I'll disable and reenable the network device!".
A few moments later: "Oh. Bugger.".
Shortly followed by an embarrassing call to the client's IT department to ask them to reenable the network device on that machine...
Yeah, this is common. Don't modify firewall rules, fuck with interface settings and addressing without another way in.
I was on my way out the door at my first job, and an SEO person stopped me and asked me if I could fix something. I just needed to delete a single row from a database. Easy enough. I get as far as “DELETE FROM table_name” and for some unknown reason run the damn thing without a WHERE. All the rows, gone!
We did have nightly backups but the guy with access to them was notoriously hard to get the attention of and lived in a different state. I was on my way out the door, and now I ended up being stuck there for another few hours trying to get ahold of devops.
At the company I am at now, we had a devops person whipe our multi TB CDN on their first day trying to make an improvement to our asset sync script. It took probably 2 days to get the whole thing reuploaded.
We did have nightly backups but the guy with access to them was notoriously hard to get the attention of and lived in a different state. I was on my way out the door, and now I ended up being stuck there for another few hours trying to get ahold of devops.
At the company I am at now, we had a devops person whipe our multi TB CDN on their first day trying to make an improvement to our asset sync script. It took probably 2 days to get the whole thing reuploaded.
> We did have nightly backups but the guy with access to them was notoriously hard to get the attention of and lived in a different state.
Read up on "bus factor": https://en.wikipedia.org/wiki/Bus_factor
Read up on "bus factor": https://en.wikipedia.org/wiki/Bus_factor
Always write DELETEs as SELECTs, and once you run the SELECT and it returns the rows you want, then swap out the SELECT * with a DELETE.
I have a couple of SQL habits along these lines that I haven't seen others do.
If I'm typing an ad-hoc DELETE or UPDATE statement, I'll slap the WHERE keyword on the end of the main command line. For example:
I also tend to wrap destructive DML in comment blocks. So in my query editor, I'd actually have:
EDIT: I've actually never made a catastrophic mistake of this nature, but there have been some very close calls. And I've seen it happen several times, from excellent programmers who just made a mistake.
If I'm typing an ad-hoc DELETE or UPDATE statement, I'll slap the WHERE keyword on the end of the main command line. For example:
UPDATE items
SET
active = 0,
deleted_at = now() WHERE
category = 'Foo'
DELETE FROM items WHERE
category = 'Foo' AND
customer = 123
I also tend to wrap destructive DML in comment blocks. So in my query editor, I'd actually have:
/*
DELETE FROM items WHERE
category = 'Foo'
*/
EDIT: I've actually never made a catastrophic mistake of this nature, but there have been some very close calls. And I've seen it happen several times, from excellent programmers who just made a mistake.
That's good advice in general, not just for SQL.
For example, I often write my 'rm' commands as 'ls' at first and then swap out the command once I know it gets the right thing.
For example, I often write my 'rm' commands as 'ls' at first and then swap out the command once I know it gets the right thing.
I count to five before hitting enter. I have stopped on 3 or 4 a number of times. 5 seconds is frequently enough time for your brain to catch up with your fingers. And nobody is going to notice it took you 15 seconds to do something instead of 10.
Yep. I do that too. Along the same lines I always do an 'echo' before I 'find -exec' anything.
Yes, I learned this from a sysadmin in the early 90s and did this when learning to use Unix and having used the technique in various other contexts (e.g. database DELETEs), I am confident this one weird trick has saved me from catastrophic mistakes countless times.
Also, open a transaction first.
Unless, like me, you then highlight the statement without the WHERE clause and run it, realizing your mistake a split second before you hear the key click...
That works halfway for UPDATE statements as well. At the very least you can see which rows you’ll be updating.
Another tip is to write the WHERE clause and SET clause prior to filling in which table is being updated. That way you can’t accidentally run something prematurely.
Another tip is to write the WHERE clause and SET clause prior to filling in which table is being updated. That way you can’t accidentally run something prematurely.
It's like gun safety. All the rigor feels stupid until you hear stories about people not following the rules.
That is really good advice. I have also found that it is good to do this with file delete commands (ls before rm).
It sounds silly, but that makes a lot of sense. Lots of tips for me to take away from this thread. :)
That's a really great idea.
I did this on the users table in production but I was fortunate enough to have started a transaction. I was just about shaking when I typed 'rollback'.
The fact that I was entering a delete -- instead of setting to inactive and doing it in production from the command line should give you an idea of how likely the backups would have been effective. This was in '97, though when everything was seat of the pants during the full internet craze.
The fact that I was entering a delete -- instead of setting to inactive and doing it in production from the command line should give you an idea of how likely the backups would have been effective. This was in '97, though when everything was seat of the pants during the full internet craze.
This 'default' behaviour strikes me as one of SQL's worst faults.
If you want to delete all rows, you should be made to explicitly type out that intention.
If you want to delete all rows, you should be made to explicitly type out that intention.
I'd say require a WHERE on deletes. If you want "WHERE true" that's fine, but require a WHERE.
If you start the MySQL Monitor with --i-am-a-dummy, this is enforced automatically
Interesting, thanks.
As far as I can tell, there's nothing like that in Postgres.
As far as I can tell, there's nothing like that in Postgres.
I worked at a public company where write access to the production databases was required to go through an intranet site which logged the query, required a code review id, and limited the number of rows that could be deleted. It was more or less for sox compliance reasons, but it provided a nice bit of safety net around dangerous queries too.
It slowed things down, but the forced code reviewer process saved me once in a big way.
It slowed things down, but the forced code reviewer process saved me once in a big way.
Back in the nineties, at a previous employer, I wrote a code generator that read a kind of DSL from stdin and spat out C source to stdout.
Having written the initial version and got it to compile, I thought I should test it before checking it into RCS for the first time. So I ran it with some simple input and piped the output to something like program.c. At that point I remembered that the code generator's main source code file was also named program.c. To my horror it had overwritten a large part of its own source.
I remember staying in the office until about 2am to re-write the lost code, but my boss never found out.
Having written the initial version and got it to compile, I thought I should test it before checking it into RCS for the first time. So I ran it with some simple input and piped the output to something like program.c. At that point I remembered that the code generator's main source code file was also named program.c. To my horror it had overwritten a large part of its own source.
I remember staying in the office until about 2am to re-write the lost code, but my boss never found out.
A couple years ago, I was writing a script to run a codesigning system (watch an input directory, sign the file, move to output directory). I had been working on it for a couple days. I got confused with my filenames, and I deleted the script, thinking it was a temp file I'd just created.
Luckily, it was Python, which compiles to a bytecode representation when you run the script, and I had done a test-run of the script recently. After a little research into the available tools, I was able to extract the original source (and comments!) from the pyc file. It was a simpler solution than any of the un-delete utilities that I found.
Luckily, it was Python, which compiles to a bytecode representation when you run the script, and I had done a test-run of the script recently. After a little research into the available tools, I was able to extract the original source (and comments!) from the pyc file. It was a simpler solution than any of the un-delete utilities that I found.
I`ve done that on a national coding competition. Not fun at all, being timed and rewriting your program with itself at the last 30mins of the competition.
I've trained myself to always use >>.
set -o noclobber
Late 90's, I'm lead dev for a large site. Company hires a DBA. We partner with a VERY large content provider. I'm awake for 48 hours buttoning things up, last minute stuff. I finally collapse from exhaustion only to be woken up at 7am on - go live day.
Nothing works.
A few moments debugging reveals that all the column names in a key set of our datasets have been renamed. No time to waste, I rename them all back as quickly as possible. We go live, no one is the wiser.
We contact the DBA and explain the folly of his decision.
He then proceeds to rename everything AGAIN on a live system. No we didn't shoot him lengthwise, but we did fire him with some prejudice.
Nothing works.
A few moments debugging reveals that all the column names in a key set of our datasets have been renamed. No time to waste, I rename them all back as quickly as possible. We go live, no one is the wiser.
We contact the DBA and explain the folly of his decision.
He then proceeds to rename everything AGAIN on a live system. No we didn't shoot him lengthwise, but we did fire him with some prejudice.
>We contact the DBA and explain the folly of his decision.
>He then proceeds to rename everything AGAIN on a live system.
What could his reasoning possibly be? Did he not understand your explanation?
>He then proceeds to rename everything AGAIN on a live system.
What could his reasoning possibly be? Did he not understand your explanation?
We later found he was acutely OCD and a perfectionist, which I've since come to find is common in DBA's. He HAD to change the column names. He quite literally couldn't stop himself. Changing it in dev, QA and staging wasn't enough for him, it was an all or nothing proposition.
incompetence
Fortune 500 company, Junior dev. Tried to log into our staging database, which has the same username, but different password than production.
Tried 3 times, got the locked out error. Got frustrated, said I'd fix it after lunch, went out to eat, came back about 3 hours later after a doctor's appointment.
My boss was waiting for me at the department door and I got sent back to his office for a stern talking to. I had locked out the production user from the database, causing every app in the company to go offline. I was then told to read aloud back all of the projected sales losses from that day as well as write a bunch of letters for our top sales people who couldn't make sales that day.
Wasn't fired (thankfully). Made it to a senior lead, so if I had to say anything, it's that the punishment really made it really drive home and I _never ever did it again_ . :)
Tried 3 times, got the locked out error. Got frustrated, said I'd fix it after lunch, went out to eat, came back about 3 hours later after a doctor's appointment.
My boss was waiting for me at the department door and I got sent back to his office for a stern talking to. I had locked out the production user from the database, causing every app in the company to go offline. I was then told to read aloud back all of the projected sales losses from that day as well as write a bunch of letters for our top sales people who couldn't make sales that day.
Wasn't fired (thankfully). Made it to a senior lead, so if I had to say anything, it's that the punishment really made it really drive home and I _never ever did it again_ . :)
I have a hard time seeing how this was your fault. It seems like a poorly designed system is the root cause, not you.
I agree with nieksand... badly designed system. It's better for companies to recognize these design flaws and fix them, rather than blame employees ignorant actions.
I dont know. Ive seen systems before where anyone could make catastrophic 'modifications' like what you describe. I tend towards blaming the system construction over the junior developer.
What a sadistic and irresponsible boss.
That time I had to update the users table so users belonging to 'department A' should now belong to 'department B'. I wrote that really simple update sentence and even diligently tested it on one of the pre-production servers. Then I copied & pasted it into the production console just to notice that I hadn't copied the WHERE clause of the sentence and now all of the users belonged to the same department.
The fix was not painful, just restore de user table from last night's backup. but I felt a little ashamed when i had to explain the database administrator what has happened.
From that day, I keep autocommit always off.
The fix was not painful, just restore de user table from last night's backup. but I felt a little ashamed when i had to explain the database administrator what has happened.
From that day, I keep autocommit always off.
Tried to get around a firewall for a windows remote desktop, changed the port number through registry, logged out, could not log back in. Was not able to change it either, detached the AWS volume, attached it to another device, could not access the registry to change the port back... I could see the screenshot, machine is running but couldn't remote in.
Doing a recursive sudo chown of the /etc folder, locked myself out of that ubuntu server.
Wrote a parser that had a condition that would not end, would watch the server spike to 100% usage.
Doing a recursive sudo chown of the /etc folder, locked myself out of that ubuntu server.
Wrote a parser that had a condition that would not end, would watch the server spike to 100% usage.
My big whoops was remotely bricking a server on top of Mt Haleakala.
I needed to update MySQL to take advantage of some new feature that young, eager me just _needed_ to use. Well, the package repo didn't have that version, so no worries, I'll install from source. Well that required some newer version of a core utility, so no worries, I'll just `yum update` that. No dice. No worries, I'll just force remove and install the newer one.
Well, young, eager me didn't realize essentially every command relied on that core utility, so although I had a prompt, I couldn't even `ls`, much less `yum install`.
Our site technician had to take an OS disk up with him next time he made the 2+ hour drive to the top. Luckily we did have a KVM set up so it didn't require a site visit from me (or is that unlucky...it would have been a trip to Maui!).
I needed to update MySQL to take advantage of some new feature that young, eager me just _needed_ to use. Well, the package repo didn't have that version, so no worries, I'll install from source. Well that required some newer version of a core utility, so no worries, I'll just `yum update` that. No dice. No worries, I'll just force remove and install the newer one.
Well, young, eager me didn't realize essentially every command relied on that core utility, so although I had a prompt, I couldn't even `ls`, much less `yum install`.
Our site technician had to take an OS disk up with him next time he made the 2+ hour drive to the top. Luckily we did have a KVM set up so it didn't require a site visit from me (or is that unlucky...it would have been a trip to Maui!).
Our 3-year-old production server hard drive, containing all assets and a database for multiple customers, died. No backups. For a few hours I was picturing not only the anger of our customers, but the demise of the whole company. Luckily the other drive in the RAID was OK and we didn't lose any data.
Probably one of the stupidest, but also the most personally effective way to learn anything.
Probably one of the stupidest, but also the most personally effective way to learn anything.
Account created an hour ago, and the only comment from this account is this one you’re looking at (which seems fine by my standards), and it’s insta-dead? I vouched for the comment, but am I missing something or is HN’s juvenile shadow ban broken again? (IP ban, maybe?)
Speaking of backups :)
We had a very large mission-critical system that we developed and hosted for a fortune 50 client.
Long story short - prod DB with tons of records took a crap, proper RCA was never done I suspected HW failure of some sort.
The kicker was - the client grudgingly OKd the restore understanding that they will loose few hours worth of data. BUT. When a DBA attempted to restore - the backup was corrupt lol. They went back day by day - all of the recent backups were corrupt, the "freshest" working backup was about a month old. Some heads rolled as a result.
The kicker was - the client grudgingly OKd the restore understanding that they will loose few hours worth of data. BUT. When a DBA attempted to restore - the backup was corrupt lol. They went back day by day - all of the recent backups were corrupt, the "freshest" working backup was about a month old. Some heads rolled as a result.
The age-old story: if you don't test restore your backups, you don't have backups.
After some bloopers early on in my career I developed a systematic "back it up" reflex if there's even the remotest chance of something being lost/broken. This has saved my bacon on countless of occasions... normally from problems I least expected.
I worked for a company once where a "consultant" was brought in to look at our "production efficiency". Some big wig came up with the idea that we should make sure we are using our production servers effectively. Not to many, not to few. We really only had 3 servers (Jerry, Kramer and Newman). Newman took a daily beating running some very large databases. It held firm though.
The "consultant" decided to move a database to its one machine. During this transition this person started a backup and then whacked the production database while the backup was still running because they felt the backup had "probably already backed up the database since it has been running for hours".
Dead in the water. Restored from a week old backup to the dismay of a lot of people working there whom had lost critical pieces of their work.
The "consultant" decided to move a database to its one machine. During this transition this person started a backup and then whacked the production database while the backup was still running because they felt the backup had "probably already backed up the database since it has been running for hours".
Dead in the water. Restored from a week old backup to the dismay of a lot of people working there whom had lost critical pieces of their work.
You're likely going to mess up sometime. Have a backup plan.
I too have "restored" a database to the wrong server (production was now development). This was a member list for an arts organization... After about 20 minutes of panic (what is going on....) I remembered I had set daily backups. whew.
I too have "restored" a database to the wrong server (production was now development). This was a member list for an arts organization... After about 20 minutes of panic (what is going on....) I remembered I had set daily backups. whew.
On the first team I worked on at [corp] we had a corner of one of the bigger labs set up as a test environment, with a row of headless PCs that were used for simulating deployment scenarios. Sitting at the end of that row was a nondescript beige box that was barely distinguishable from the test boxes, except for being a bit older, that had our team's internal website, all of our documents, scratch file shares, databases, etc. on it. Of course there were no backups. At least two times while I was there, they hired temp workers and on day one gestured in the general direction of the shelf full of machines and told them something like, "OK, you can get up to speed by reimaging those boxes for this week's test pass..."
During the beta for a product, we were working on sending out daily summary emails to our users. We had a cron job which should run at 8:00 AM in the morning which would process the queue and summarize all activities into a single email.
It being a very long day, I deployed the job with the cron expression * 8 * * * instead of 0 8 * * * , which meant it ran EVERY minute starting at 8:00 AM instead of just once.
The next morning I woke up to a flurry of text and emails saying customers had complained they received emails about a hundred times. Thankfully, we had a pause in the job which meant it went out only a hundred times and not thousands as one would have expected!
It being a very long day, I deployed the job with the cron expression * 8 * * * instead of 0 8 * * * , which meant it ran EVERY minute starting at 8:00 AM instead of just once.
The next morning I woke up to a flurry of text and emails saying customers had complained they received emails about a hundred times. Thankfully, we had a pause in the job which meant it went out only a hundred times and not thousands as one would have expected!
Oh yea, I've done that too. These days I use systemd timers, which are much less prone to that sort of error.
We were group debugging a crash that happened while using our API for a network device for NetWare under heavy load. We weren’t sure if the problem was our code or the IPX driver we had written for the particular NIC, so we were swapping in every NIC we had in the lab, and running some samples. Actually, I was doing the swapping. Turns out, the one time I forgot to power down was removing an NE-1000 (cheapest NIC available) and swapping in a 3Com 3C505, worth about $900 in 1988 dollars. The NE-1000 was fine, but the 3C505 was toasted. The PC was fine. The boss totally clowned me with some mean mugging, then smiled and said it was ok.
We had weekly releases to production. during my first month I checked in some code without yet fully understanding the spaghetti'ness of the 20 year old codebase.
it turned out that our beta testers found a lot of cases that were broken on Friday when the release was next Monday. The problem was that a ton of other classes pointed to the one in which I made changes and altered the state..
Worked through the weekend with my teamlead and got it fixed somewhere on sunday afternoon.
good bonding experience though. All had a laugh about it the next week. :-)
it turned out that our beta testers found a lot of cases that were broken on Friday when the release was next Monday. The problem was that a ton of other classes pointed to the one in which I made changes and altered the state..
Worked through the weekend with my teamlead and got it fixed somewhere on sunday afternoon.
good bonding experience though. All had a laugh about it the next week. :-)
As for "Why So Many Dropped Production Databases?", you can always configure ssh with a combination of LocalCommand or use tmux to change the background of the terminal, when you are on a production server.
I did a blog post about it [1].
1: http://www.drinchev.com/blog/ssh-and-terminal-background/
I did a blog post about it [1].
1: http://www.drinchev.com/blog/ssh-and-terminal-background/
Here is a fun one for you:
In Git, a push-force used to (or at least how I remember it) only force-push your current branch. I force-pushed on a branch... and it force-pushed master. Thankfully no one had pushed to master since I last pulled from it.
Here is the fix:
``` [push] default = current ```
Or you can just, not force-push... or be explicit, which I do now. I use force pushes for rebasing and --amend. I am one of those weirdos who enjoys clean, readable git repos.
In Git, a push-force used to (or at least how I remember it) only force-push your current branch. I force-pushed on a branch... and it force-pushed master. Thankfully no one had pushed to master since I last pulled from it.
Here is the fix:
``` [push] default = current ```
Or you can just, not force-push... or be explicit, which I do now. I use force pushes for rebasing and --amend. I am one of those weirdos who enjoys clean, readable git repos.
Stop mucking around in production. Stop. Seriously, stop.
If you have never screwed up anything in production, no one at your org trusts you enough to give you production access. Which is ok and great for you, but at the end of the day SOMEONE has to have production access. You can write a devops template that wipes out data just as easy as you can sudo rm -rf /
I am much more terrified of Terraform than myself, especially `terraform destroy`. I mean, a human can only do so much so quickly but a script could kill everything semi-instantly.
I look at `terraform plan -destroy` and it still terrifies to destroy an internal service.
I look at `terraform plan -destroy` and it still terrifies to destroy an internal service.
We actually had a production outage due to a `terraform destroy` typed in production instead of dev.
I say that all the time. You automation can be just as dangerous as prod ssh access.
And in some ways it is worse..
I remember at least one prod incident where a script made it out into prod and wiped out 100s of servers. Something like it had rm -rf /$SOME_ENV_VAR which was set in QA but not prod. Opps....
I remember at least one prod incident where a script made it out into prod and wiped out 100s of servers. Something like it had rm -rf /$SOME_ENV_VAR which was set in QA but not prod. Opps....
Imo, it's not about trust, it's about rigor.
Yah? Good luck with that
Another thing you can do to minimize the risk is limiting what your database user can do.
For instance, if you usually just need to view content. Don't let him truncate / delete / do other destructive operations.
For instance, if you usually just need to view content. Don't let him truncate / delete / do other destructive operations.
Once I refactored some stringly typed C# code to use an enum with values Prod and Test instead of the two strings "Prod" and "Test".
I was relying on expressions like
Sadly some parts of the codebase weren't very idiomatic and had some
I was relying on expressions like
mode == "Test"
Sadly some parts of the codebase weren't very idiomatic and had some
"Test".Equals(mode)
>stringly typed
I'm going to steal that one.
I'm going to steal that one.
I once deleted the entire contents of /bin and /sbin on an HPUX box ~2004 with the HPUX servers origin circa 1996. Live ssh session: non-standard location. scp from a convenient near-replica and voila. Without the near-peer I would have quit and never went back to IT.
We had a good one recently where / got bind mounted to be used as a container rootfs in a quick and dirty test, then we decided to `rm -rf` the directory containing the bind mount. A clever way to get around pesky --no-preserve-root not being the default.
At my first job, guy in my group once su'd to system user that ran our infrastructure; and typed, cd && rm -rf * and went home.
I was sitting at my desk watching service after service slowly disappear in utter confusion, then hours spent cleaning...
I was sitting at my desk watching service after service slowly disappear in utter confusion, then hours spent cleaning...
... Why would someone do that? Was it ineptness? Just being tired? That seems so blatant.
I love this comment thread, there's a cool app idea here if done right!
One for local dev and another for a production server running a trading system for around 50 users. Naturally for production I was logged in as root.
I typed rm -rf to clear out my dev folder only to figure out I just deleted the root folder and all sub folders on a production machine.
I recalled a piece of advice a university friend had given me. "Never admit to anything."
I walked into the server room, switched off the PC and restored it from a backup tape that was a day old. 2 Hours later the machine was back on.
No one ever noticed, and I never mentioned it either.
Moral of the story - Don't let anyone give you access to production servers.