Man deletes his entire web hosting company with single line of accidental code

By midian182 · 37 replies
Apr 15, 2016
Post New Reply
  1. In what could be the ultimate example of having “one of those days,” a man has managed to delete his entire company by mistakenly running a single piece of code.

    As reported by The Independent, Marco Marsala, who runs (or perhaps that should now be ‘ran’) a small web hosting company, inadvertently ran the ‘rm –rf’ command line while performing maintenance operations on his servers.

    For those who don’t know, ‘rm’ tells the computer to remove, and the ‘-rf’ part deletes everything by force, overriding the usual safety warnings.

    The command line is often used to delete from a specific directory, but none was given because it was ran by accident.

    Marsala used a Bash script that was supposed to set the location, but a bug in the code above the line in question meant no directory was given, resulting in absolutely everything being deleted.

    Writing on the Server Fault forum, Marsala said: "I run a small hosting provider with more or less 1,535 customers and I use Ansible to automate some operations to be run on all servers. Last night I accidentally ran, on all servers, a Bash script with a rm -rf {foo}/{bar} with those variables undefined due to a bug in the code above this line.”

    Marsala went on to explain that the line of code had also deleted his backups and all his customers’ websites. “All servers got deleted and the offsite backups too because the remote storage was mounted just before by the same script (that is a backup maintenance script)."

    Forum users told Marsala that his only option was to now contact a data recovery service, hope for the best, and get a good lawyer. He was also criticized for not putting safeguards in place that would have stopped this particular line of code from running without a specified location.

    "This is not bad luck: it's astonishingly bad design reinforced by complete carelessness," wrote forum user Massimo.

    Image credit : kikovic / shutterstock

    Permalink to story.

  2. H3llion

    H3llion TechSpot Paladin Posts: 1,376   +286

    ... and that' is why ladies and gentlemen you make backups off of the main server you are hosting from.
    Hexic, psycros, wastedkill and 2 others like this.
  3. stewi0001

    stewi0001 TS Evangelist Posts: 1,681   +1,080

    Yea, he's screwed. I feel sorry for him, unless he's a jerk then karma. But I'm going to assume he's a good guy who made a mistake.
    Puiu likes this.
  4. davislane1

    davislane1 TS Grand Inquisitor Posts: 4,736   +3,757

  5. jobeard

    jobeard TS Ambassador Posts: 11,128   +982

    If it was within some script given him, the provider is liable and stupid.
    if he himself type that line, then he was stupid and had no business being the admin.

    btw: rm -I foo would have prompted for permission, eg
    Remove (unlink) the FILE(s).
    -f, --force
    ignore nonexistent files and arguments, never prompt

    -I prompt before every removal

    -I prompt once before removing more than three files, or when removing
    recursively; less intrusive than -I, while still giving protection
    against most mistakes

    -r, -R, --recursive
    remove directories and their contents recursively
    H3llion and madboyv1 like this.
  6. Puiu

    Puiu TS Evangelist Posts: 2,653   +1,092

    Aren't backups supposed to be protected from stuff like this?
  7. Skidmarksdeluxe

    Skidmarksdeluxe TS Evangelist Posts: 8,647   +3,274

    Imagine if he worked for Google...
  8. jobeard

    jobeard TS Ambassador Posts: 11,128   +982

    Protect, only in the sense of recovery
  9. Thrackerzod

    Thrackerzod TS Enthusiast Posts: 40   +30

    This story seems incredibly suspect to me. I have a feeling a lot of news sites have been trolled.
    wastedkill and gingerbill like this.
  10. Uncle Al

    Uncle Al TS Evangelist Posts: 3,326   +1,974

    Incredible ... simply incredible ..... and stupid!
  11. 9Nails

    9Nails TechSpot Paladin Posts: 1,215   +177

    Indeed. It's an old rookie mistake, but it'll serve as reminder to double check my code before putting it into production.
  12. BSim500

    BSim500 TS Evangelist Posts: 388   +661

    Moral of the story - the only true backup is an offline one (not just offsite). Not just vs accidental deletions, but malware, hacking, disgruntled employees, etc. Cloud storage is simply another layer enhancing an existing backup solution, not a replacement for it.
  13. Teko03

    Teko03 TS Evangelist Posts: 414   +186

    Came to say just this. With 1500+ customers, he should have had an off-site/rotating tape backup plan implemented hundreds of customers ago.
    psycros and H3llion like this.
  14. Camikazi

    Camikazi TS Evangelist Posts: 925   +284

    He did but that script mounted it before it ran the rm command, it was a very, very bad script.
  15. andrewdoyle88

    andrewdoyle88 TS Booster Posts: 113   +95

    TheDreams and H3llion like this.
  16. daThomas

    daThomas TS Rookie

    I'm gonna say at the point you have 100 website hosting customers, you should be doing weekly offsite backups with the drive being stored in a safety deposit box.

    Also, gotta empower customers, and encourage them to make their own backups.
  17. TheBigT42

    TheBigT42 TS Addict Posts: 155   +82

    Only if you don't run rm -rf on them and don't keep any offline backup disks / tapes.
  18. TheBigT42

    TheBigT42 TS Addict Posts: 155   +82

    Amazon has some great cloud storage option that he could have backed up to
  19. VitalyT

    VitalyT Russ-Puss Posts: 3,663   +1,949


    IAMTHESTIG TS Evangelist Posts: 1,255   +454

  21. mailpup

    mailpup TS Special Forces Posts: 7,182   +469

    It does sound like the kind of story that later turns out to be a hoax doesn't it?
    gingerbill likes this.
  22. Camikazi

    Camikazi TS Evangelist Posts: 925   +284

    It does, but something in me wants it to be true as mean as that might be.

    Edit and Update - Seems it is a hoax that is a link to a story about the guy saying it was just a joke.
  23. Fabio Turati

    Fabio Turati TS Enthusiast Posts: 28   +12

    This story turned out to be a hoax. It's explained in a newspaper article in Italian, which can be summarized like this:
    "It's just an experiment to show how little everyone knows about this"
    "That command is harmless but it looks like no one has noticed it"
    "I wanted to advertise my startup which offers to outsource the management of servers"
    "I am even writing a book on Unix Horror Stories for Dummies; anyway, this really happened to a person I know, but that was some years ago"
    Then he added that after 2006 that command doesn't do anything without the "-no preserve" option, and that Ansible prevents this kind of errors, and almost every competent administrator uses it, but it looks like that none of those who answered knows it, otherwise they'd know that what he said can't happen.

    So... Enjoy viral marketing.

    And, on a side note, I would like to point out that Server Fault, which belongs to the Stack Exchange network, just like the more famous Stack Overflow, is not a forum. It's a Questions&Answers site, and all the users stress the difference.
    Lionvibez likes this.
  24. gingerbill

    gingerbill TS Addict Posts: 233   +59

    This sounds like nonsense , or a jewish stocktake.
  25. Blake Gall

    Blake Gall TS Rookie

    Perfect :D

Similar Topics

Add your comment to this article

You need to be a member to leave a comment. Join thousands of tech enthusiasts and participate.
TechSpot Account You may also...