What was your biggest *nix blooper?

Question

So I was thinking about bloopers I've made or have seen made in a *nix systems and was wondering what others think the worst blooper they made was? I couldn't find a similar question already asked but I can't believe this isn't a copy question but then again may users are so perfect the idea of a blooper is impossible. Fire away.

Answer 1

Anyone who answers something other than rm -rf / is lying ;)

Answer 2

First, accidentally creating a file in my home dir with the name "-f". Which made it a bitch to remove cause -f is a flag. So I figured I would do some thing like this while in my home directory:

rm -i *

And just say no to all the ones I wanted to keep.

Unfortunately, -f overrides -i ... bye bye home dir:(.

One correct solution is to use --:

rm -- -f

Oh well, lesson learned.

Answer 3

Rebooting without updating LILO. No boot disks or internet handy to figure out how to fix it...

Not quite up to rm -rf /

Answer 4

I once used

killall command

on an older Unix machine with the intention of killing all instances of command. It turned out that on this machine, the killall command killed all processes on the machine in preparation for shut down instead of the more modern version which kills all instances of a specific process. Whoops.

Answer 5

One of our sysadmins once did

rm -f *

at root, logged in as root (!).

Fortunately, not rm -rf.

So all the symbolic links disappeared, including (from memory) /bin. Which made finding /bin/sh a bit difficult when logging on. So no logons.

Then the user logged out as root (but was still logged in as the user. This was significant, although, not, I think, essential)

I used to use this as an interview question: how do you recover from this situation? (We did, eventually, one of the other sysadmins sorted it)

Solaris, in about 1992/3, if it makes any difference.

Answer 6

My answer does not involve rm in any way:

tar cvzf /dev/hda

Answer 7

Instead of running

/etc/init.d/networking restart

on a Debian box, I ran

/etc/init.d/networking stop

... whilst I was connected over SSH.

It meant a swift drive into the data centre was in order!

Answer 8

I ran wuftp on my first Slackware install.

Result = rooted :(. Fortunately the idiot who rooted me setup an IRC bot and decided to store the log file in my only user directory. It only took me a week to notice the 8 GB file sitting on a whopping 20GB drive. They were nice enough to include their IRC handle and origin IP addy in the log file.

Answer 9

Not on the scale compared to rm -rf but...

When I was learning *nix, I was bouncing between at least three very different OSes and used my habitual name for a quick and dirty test C program -- and named it test

I spent I don't know how long trying to figure out why my little Hello World equivalent wouldn't output anything. Finally, I asked my resident *nix expert and he laughed so hard he cried before pointing out that test was a builtin and ahead of my dir in the path.

Answer 10

I once had to update an important shared library - libgcc, I believe it was. I went about it in the most brilliant way possible - I compiled the new library, then cd'd to the directory where the old libgcc was (yes, the one being currently used). I then did "cp newlibgcc oldlibgcc".

What's this? Oh right, I have to be root to overwrite a file! "sudo su", then "cp newlibgcc oldlibgcc"

Instantly locked up my box, as you can imagine.

Answer 11

I was cleaning out a local directory, so:

 rm -rf *

No problem, except that it left those little .xxx resource files, so:

rm -rf .*

which apparently follows .. and heads up the directory tree (oops)

... and of course I had root privilege

Answer 12

Wrote a shell script to recursively process all files in a directory and call an external application on them...only problem was instead of calling the external app I called the shell script again. Ended up with a script that processed a directory of 1000+ files recursively calling itself and brought the server down with what could be considered a denial of service attack. I knew something was wrong and would kill the script but I ran it a half-dozen or so times trying to figure out what was going wrong.

Answer 13

Was doing an rm -r of a user's directory (that no longer worked at that place). That user had placed a hardlink in there that pointed to a group shared directory... Guess what... -r recurses through the linked directory erasing all the shared files instead of just removing the hardlink...

Luckily there weren't any changes yet that day so I could restore from the backup. Took awhile though, there was a LOT of data there...

Answer 14

On a Debian box, I was having trouble with a wonky package, and I needed to forcibly remove it. But I must have been tired, and accidentally removed dpkg instead of the other package.

"No problem" I thought, "I'll just re-add it"

Nope. On a Debian box where there is no package manager, it's sortof a "chicken and egg" scenario needing to get the package manager again. :)

Thankfully, I still had SSH access to the box (it was being remotely managed), and I was able to back up the system before doing a complete reinstall.

Answer 15

mkfs on the wrong partition.

Answer 16

Eager to fork() for the first time and having difficulty with a fork() tutorial, I incrementally commented out lines of error checking until, finally, the fork() call was executed. Inside a loop. With no error checking.

When my connection slowed to a crawl and then dropped, I assumed it was a networking problem. I tried again on another machine, one that happened to be the primary authentication server for our campus network. This time, my connection slowed and dropped and other people in the lab began having problems. Understanding dawned and my body slacked a moment.

Pulling myself together, I reported to my guru who insisted on a pompous walk to the campus network operations centre where the sysadmin, having already cut his lunch short, was duly chastised for allowing fork bombs to occur in the first place.

Humiliated and honoured in the same act, I grew a little older that day.

Answer 17

Not my biggest, but my most recent was upgrading Ubuntu to Intrepid Ibex, and leaving for the weekend. I came back and it stopped updating; I restarted the computer and I guess all the packets didn't update properly, so now it doesn't boot up correctly; if at all!

Answer 18

I changed the default SSH port on a new Slicehost VPS and then configured iptables...forgetting to open the new SSH port, which meant I was locked out of the VPS as soon as I logged out!

Fortunately Slicehost let you recreate a slice from their SliceManager Web interface.

Answer 19

Backing up some files with tar, then blowing away the filesystem those files were on.

I hadn't tested the backup, and found out when I went to restore the backup that it didn't work.

I suppose this could apply to any OS, though.

Answer 20

I did cp somefilename /dev/hdc1

Answer 21

Meh, not a huge mistake, but a pretty new one when i first started, i had been Trying out Ubuntu and had it dual booting, back then i didn't understand how the MBR worked though so... well you can see what happened.

"Well time to get rid of Ubuntu, i will just use the gparted live cd to delete the partition and give it back to windows!"

needless to say, i had to find a way to reinstall the windows bootloader when GRUB started crying to me :)

Answer 22

so I have to admit I've seen a guy do a chown on a server box as root to a mostly unused account that effectively read

sudo
cd /some/far/off/directory/from/root
chown weakling_user *.*

this mostly only brought things on the server to a halt until it was cleaned up but as part of those who felt the effects I feel it may be one of those really big mistakes.

Answer 23

A long time ago, and someone else hit the return key, but...the net result was 'rm -fr /'.

Fortunately, it was a new machine. Fortunately, they had a backup. So, it wasn't a complete disaster. But...

Needed to install some software. Needed a user created temporarily. Used menu system to create user. Since user was temporary, gave user home directory of '/'. Installed software. Used menu system to remove temporary user.

• "Do you want to remove home directory?"
• "Yes"

It takes a while for everything to grind to a halt, but /etc and /unix and /bin are very early on the list of removed files and directories.

Recovered system from backups. Created temporary user with real home directory, not "/", and reinstalled software, re-removed user, etc. Delay - 4 hours or so. Could have been so much worse.

Answer 24

I left a space in a mv command, and ended up moving / to /home/username . On a production mail server. In the middle of the business week.

Answer 25

rm -rf on my home folder during my industrial placement. It was accidental, had a * in the wrong place. Learned the value of using CVS.

Answer 26

rm -rf /tmp backfired on me one day when another sysop somehow/someway remounted an important partition to /tmp/homes and symlinked that to where it should have been ( in this case /home ).

Always fun as a junior Admin to nuke your colleges student accounts. Ever since then I use find | less before using rm -rf.

Answer 27

In OS X, I wanted to move my /users folder to an external hard drive. So I naively moved the folder over, then created a symlink on the original drive pointing to the new location. Everything from iTunes to my Word Processor went haywire. I'm still not really sure why it didn't work.

Answer 28

I edited the iptables configuration file on our Linux firewall server and deployed the change. Unfortunately I had made a mistake in the configuration and I blocked all network traffic to our company.

And doubly unfortunately, I was logged in remotely via ssh. Of course my ssh session immediately terminated. I had to call our system administrator, who was already driving home for the day. He turned around, drove to our data center and fixed it at the console.

• Lesson 1: only the IT department should have root access on critical servers.

• Lesson 2: never do anything on the firewall server remotely, even if you are the authorized IT expert.

• Lesson 3: bake cookies for the system administrator in penance when you screw up.

Answer 29

rm -r'd a dir I didn't mean to, moved libc on a live system and watched it panic(), screwed the bootloader, symlinked something to /bin/sh which was linked to /usr/lib stuff (and I wondered why init cries: /usr not mounted after init/rc). And many more I dont remember. Unix gives rope for sure. :-)

Answer 30

My biggest involved root permissions and cdrecord, back in 1997. Root was required to get real-time scheduling which was definitely required on a Pentium 166 burning CDs. The cdrecord command did not accept /dev/hdc as a valid destination. Instead you had to use direct SCSI addressing, something like (7,0,0) as I recall.

And that was the end of the NT installation on that machine, which was on the second hard drive, which made the CD-R the third device.

Luckily Linux survived.

Answer 31

A scsi controller and drive were connected during an install/migration that should not have been. When installing I was not paying enough attention and when prompted to format and use all available disk space and to auto-create partitions. I was thinking 'sure this will take less time than manually arranging my partitions'. This subsequently erased and wrote over a 9gb drive that was full of files that there were no backups for (this was when 9gb was a lot).

Answer 32

I did:

rm * ~

(deleted all files in the current directory)

instead of:

rm *~

(delete all temporary files created by my editor)

before committing changes to the repository.

Answer 33

Many moons ago, in the wee hours of the morning, I sat there with a ssh terminal to my remote webserver, and was doing some cleaning up.

There were all these archives from old websites that were no longer live and I thought I'd copy them to my local machine, burn them to a CD and free up some space. The way I had the directory was as such:

/var/www # top level
/var/www/archives # archives directory
/var/www/html # this had all the live websites

I then ran:

rm -r /var/www <tab> <enter>

Notice what was wrong ?

Ran it for a few seconds, and then it hit me - I just ended up deleting all the websites. :(

Answer 34

i love this :(){:|:&};: but i dont think this beats rm -rf /

Answer 35

I once mounted a remote SAMBA share to my home directory, thinking it would magically mount to some subdirectory within my home directory.

Basically it had the effect of rm -rf ~, except fixable with a cold boot. Phew!

Answer 36

 # source ~/.bash_history

The TAB completion got me. I wanted to source ~/.bashrc and sourced ~/.bash_history instead (root didn't have a .bashrc)

Answer 37

I had a nasty keyboard with a big return key that extended to where ~ was meant to be, so when I tried to do "rm -rf ~" I lost a lot more than I intended.

It could have been a lot worse, I wasn't in / at the time.

Answer 38

A former college (no he wasn't fired) misconfigured a script made to send text messages to mobile phones. This resulted in the flooding of a handful customers' mobile phones with about 100 000 text messages before it was noticed and stopped.

Answer 39

Configuring a mail redirection daemon into an infinite loop:

The daemon didn't simply forward; it read the email, made a comment, and then created a new email.
Thereby, not being flagged as a dupe, and each iteration was larger than the previous one.

The resultant email was not only sent to address from which it was sent from,
but bcc'ed to others in a similar fashion.

:s

Answer 40

I recently could not figure out why my test program (compiled as "test") could not run on ubuntu. Read all about it here

Answer 41

I was trying to identify a problem on a remote customer machine and wanted to see the local routing table, so I issued a route -v to get a verbose output. Unfortunately, I hit the -f instead which flushed the table and severed my connection.

Why use two keys so close to each other for "get verbose output" and "get rid of everything" :(

Answer 42

My worst mistake was in writing a script that sets up a bunch of directories based on the input.

One of the commands ran this:

rm -rf $4/$5

The problem is that later on I forgot how many parameters the script took, so I just called it with no parameters without thinking about it.

So I ended with this... run as root of course:

rm -rf /

Lesson learned: Always test sanitize inputs, and have a default function to print usage!

Answer 43

This wasn't me, I swear.

killall - on a Tru64 cluster.

Answer 44

My first linux install....

I installed on a partition, and then was reading along to where it suggested installing another version of some file having to do with command processing (long ago, can't remember the details). In those days, I had a habit of never typing "mv A B" while logged in as root, after having gotten the source and destination mixed up, so I typed "rm B" automatically, to be followed up with "mv A B". The second command never made it.

Fortunately, since I had just installed, I lost nothing but a bit of time reinstalling.

I was young and foolish back then. I'm older now, anyway.

Answer 45

Ancient Unix machine where you had to be root to write to a tape.
"tar cvf /dev/rmt0" every night to write the data, take the tape out store it carefully.
Discover that the command works perfectly without a tape loaded!
Discover a large file called "rmt0" in the /dev directoy.
Discover that the tape drive on this version of Unix is called "rst0"

Answer 46

About 10yrs ago I was on an unused sever at my job, I forget the flavor of *nix, but I wanted to copy some files before I updated something, and made a typo that looked like

cp /usr/lib /usr/lib

which basically copies the folder over itself, which turn every file into 0 bytes, and because it was an OS folder, instant system death.

Answer 47

My biggest one has probably been fsck -n on /home, a live ext3 filesystem. Hey, it's not going to make any changes, it should be safe, right? Apparently not. Forced me to reboot; the fsck on reboot "fixed" the damage by (a) noting every inode was not deleted, yet had no name and thus linking every inode to lost+found. With such useful names as the inode number. But I keep backups, and it was just my personal machine, so it wasn't that bad.

I once had fun with rm * .o (note the typo) a long time ago, but due to CVS, it wasn't that big a deal either.

Took down a small ISP once — including my connection to the router — once by breaking OSPF horribly. Managed to get enough static routes in to fix that w/o a drive to the colo.

Answer 48

root: rm /dev/null

It really did remove it!

Answer 49

I was trying to delete all of the temporary backup text files in /etc/ such as "blah~". What ended up being typed was:

cd /etc
rm -rf * ~

And of course there was no backup of the /etc folder. Darn space bar!

Answer 50

I was working with a broken Fedora Core 6 installation and wanted to reinstall a broken package. Unfortunately, one of it's dependencies was libstdc++.so.

So, I tried to remove the package that contained that file and in doing so, I wasn't able to run any new commands to reinstall it...

Answer 51

While testing the security of my IRC bot, I was so eager to see what would happen if I'd push it to the max. Those lines are another user's log:

<LiraNuna> !fq ["shell exec 'sudo rm -rf /'"]
* Sintia has quit (Remote host closed the connection)
* LiraNuna has quit (Remote host closed the connection)

Answer 52

I wrote an automated script to build a bootable flash drive from a source tree on the hard disk. At the end it installs LILO on the drive.

One day I somehow screwed it up and it trashed the bootloader on my main hard disk. No problem, I thought, I'd just go and fix it with a livecd after I did the flash drive. So I got in there and saw LILO had stored a backup of the MBR in /boot/. I dd'ed it onto the hard drive and rebooted. That's where I found out that was the MBR from the flash drive.

Answer 53

I had worked for several days on an important project and, tired, I just noticed that the machine had no swap activated. So I 'mkswap /dev/hda1' where hda was the /home partition...

Answer 54

A former manager of mine got curious when he saw the strange string :(){ :|:&};:& on the whiteboard in our office and asked me what it was about. I told him it was a shell command to hog memory on unix systems. He wouldn't believe me it would actually parse, so he typed it in on our Solaris box.

---

It took him 15 minutes to catch the unix system administrators over the phone. It took them 15 minutes to realize they wouldn't telnet/ssh in, even as root. It took some additional 30 minutes for the box to reboot and start accepting users again.

Their conversation on the phone was something.

 - "What the hell is going on?  What have you done?!"
 - "Errr... we've got 'some process' gone wild, could you please kill it?"
 - "Trying to, but... hey!  I can't even connect to... does it even ping anymore?"

Countless mandays lost. The next week, users had process and memory capping.

Answer 55

A friend of mine was in the administration team for the students' network. He was into computers, but a bit new to that thing called "Linux".

He was logged in on the server (let's call it darkstar), working on files mounted over Samba from the Windows server next room. Once he was through with his stuff, he decided the mount point wasn't canonical enough by his standards, he should move it someplace else. "Easy! I'll just unmount it from there and remount it in the new location."

Except unmount isn't how Linux calls it.

$ unmount /mount/point
bash: unmount: command not found
$ rmdir /mount/point
rmdir: `/mount/point': Directory not empty
$ rm -rf /mount/point
rm: cannot remove `/mount/point': Permission denied
$ su -
Password:
# rm -rf /mount/point

Turns out, he knew just enough to be dangerous. The funny part, retrospectively, is that it took long enough for him to get suspicious and think "that's not right". But he didn't know about ^C yet.

Answer 56

I was modifying a network script for an embedded system where I was only able to send one command at a time. It was a painful^Wlearning experience in playing with sed and awk. Since I was unsure the resulting file would be correct I piped all my changes to a temporary file which I downloaded and checked for correctness after all was done. The file contents were all okay so I moved the script to the correct place and rebooted the system.

After waiting for what seemed like forever it started to dawn on me that the system was probably not coming back up again so I tried reproducing it locally which is what I should have begun with anyway. Turned out that the script needed the execute flag set and since I just piped changes into a temporary file and then moved it to the appropriate location it did not have the execute flag set.

Big oopsie and cost us some money since we had to send out a technician to replace the entire box. At least I learnt never to modify remote systems without trying it locally first.

Answer 57

A variant of the killall stories. A mate was in the habit of running

kill -9 -1

to terminate all his running processes and logout. It became so much of a habit that he did it while logged in as root on a clients production machine.

Answer 58

Network administration and IRC pissing contests don't go along that well.

During school holiday, two members of the student network's administration team were back home (separately). They both had modem access, and were chatting with one another on IRC. The IRC server was the students' too, hosted on the campus. For some reason, they started comparing bandwidths. The easy way to measure would be to ping flood the other one until either dropped off IRC.

That's until the younger one thought it could be a good idea to use the IRC server's better connection.

I was physically on the campus at that time, and I got woken up by a phone call from the older one. "Hey, err... would you mind ssh'ing to darkstar, here's the password, and please kill that ping -f process for me."

As it turned out, neither of them was first knocked off the IRC server that night. On the other hand, the IRC server had dropped the whole local network + campus + school off the net.

Answer 59

Tried to delete all coredumps in my homedir

find . -delete -name 'core'

Answer 60

Deleted a symlink to my /usr folder, but did rm symlink/ instead of rm symlink due to tab completion. This was before I added a regular user to my freshly installed Linux box. Spent the next week using the computer and fixing problems as they cropped up. Eventually got sick of it. Lesson? Watch what you do in root. Always.

Answer 61

I'm a Gentoo user. I was uninstalling a package and wanted to remove all traces of it. So I got a list of what files "belong" to the package via a tool called equery. I then took the contents of the file and fed it to rm. After it took a while to complete, I checked the file list, something I should've done earlier. It had started to delete the entirety of /etc and /usr. Fortunately my /home was untouched, and I could get all my data back. Also fortunately it was my desktop. Installing Gentoo takes forever though.

Answer 62

Was a long time ago, so I forget the details. Was on Solaris. The disks on Solaris were partitioned in such a way that one of the partitions represented the whole disk (partition c? I forget.) Anyway, one day, somehow I managed to forget this important piece of information, and poking around on a system, I was thinking, hey, what's this huge partition that's not mounted or anything? Hey, I should make a filesystem on it, no sense having disk space go to waste. Uh, hmm... the system seems to be getting a bit sluggish, what's going on... realization of what I've just done dawns... "oh, crap."

A coworker once came by, "you'll never believe what I almost did! I was adding a disk to the system, and was about to create a filesystem, and I almost ran newfs on the wrong disk!". "Whew!" He heads back to his office, where he had left the mistyped command, sitting there, waiting for him to backspace over it, or type ctrl-U, and so he sits down and his chair and uh, he hits "enter" a couple of times. Oops.

Answer 63

Within the second week at my previous internship, I ended up typing the following command (with root privileges, of course) on one of our primary test servers:

cd /bin; rm *

Things didn't work so well after that.

Answer 64

Once, my antispam server detect a BIG amount of mails incommings.. I blocked out in the firewall.. 20 minutes after, my boss tell me: Hey, you are blocking our server!! Ouch..

Answer 65

I had a python script in a directory that created log files. I needed to delete these log files:

rm -rf *;log

Oops I meant:

rm -rf *.log

I pressed on the SHIFT a little bit too slightly.

So I deleted everything in that directory: the log files and the python script.

I didn't wanted to recreate it from scratch so I had to grep the directory to retrieve it based on some strings I remembered in my script. Something like:

dd if=/dev/sda1 | hexdump -C | grep 'some string I remembered'

Answer 66

In my attempt to remove old log files, I created a cron job:

"find . -mtime 30 | xargs rm"

Forgot the "cd " first and it started removing EVERY file on the system that was older than 30 days since cron starts at /.

Answer 67

I had my desktop filled with terminal windows, some local and some remote ssh connections, logged in as root in most of them, and with '#' as prompt in most of them. I had to reboot my box, so I grabbed a terminal and typed reboot. When it said 'Connection to somehost closed by remote host' I realized I hadn't booted my own machine, but our company's authentication server for some thousands of dialup customers. It was an old Solaris box that used about 15 minutes to boot...

Answer 68

In college for a final project, I did

rm * ~

the day before my project was due. Spent the whole night re-writing the code. Funny thing is I think the code turned out better than what was deleted. This is exactly why they should have used source control for assignments! or I should have been using it myself :)

Answer 69

This was many years ago, and the first time I ever tried Linux. :)

I'd installed Red Hat 9, but I was a bit annoyed that I had to sudo to access a number of files (keep in mind, first time, experimenting, whatnot). Eventually, I wanted to change that.

So, I switched to root, went to /, and did a chmod -R 0777 *.

That didn't go so well - though strictly speaking, the operation itself succeeded. Turns out, however, that a lot of things - including the X server - won't start if the permissions are wonky. Even getting to a shell was a bit of a problem.

Fortunately, I hadn't really done much with the install, so it was just a matter of reinstalling. I've learnt my lesson since then.

Answer 70

managed to type kill 9 3456 , to get rid of a stuck process of ours. I meant kill -9 3456

I don't know which procees had pid 9 on SunOS but apparently it was important - so was the services running on that box unfortunately.

Answer 71

> cd /my/latest/version/of/project's/code
> tar -zxvf /my/oldest/version/of/project's/code/code.tar.gz .

Loss: 5000 lines of C/GTK+ code. This project never saw the light of the day.

Answer 72

Switching between ssh windows to do a chomod 0666 in a directory the realising I was in the wrong window (/var as root) 8(