The backup predicament… not enough drives!

Information and news about the website will be posted here.
Post Reply
Andy
User avatar
Administrator
Posts: 12815
Joined: Fri Aug 18, 2006 11:47 am
Location: United Kingdom
Contact:

The backup predicament… not enough drives!

Post by Andy »

The backup predicament… not enough drives!

As you all may well know, backing up these days is essential and everyone should be doing it. Sadly not everyone does. However I do, but it came to a point today where I ran out of space on one of my 1TB drives. I could just start storing data to another drive but I wanted to keep things neat, but with the way I was backing things up and trying to keep redundancy, it made things difficult. In the end I had to juggle drives and data around, and came up with a solution, but it resulted in a slightly lower redundancy level than I originally had. I’ll try and walk you through how and why I did it now, so you can see that even the seemingly complex problems can have simple answers.

I’ll start by telling you how the drives were laid out and labelled before I changed things.

C:\ 80GB (boot) ~10GB used
M:\ 500GB (www_data) ~20GB used
N:\ 500GB (Backup) ~250GB used
X:\ 1TB (Data) ~25GB used
Y:\ 1TB (Betas) ~750GB used
Z:\ 1TB (Media) ~910GB used

As you can see, the data is split up “neatly” so I know where to find it, but it’s also redundant in that the main data is backed up and split across several drives.

C:\ backs up to N:\
M:\ backs up to N:\
X:\ backs up to N:\

Now, as you can see from the first table, X:\ is only using 25GB of the available 933GB the 1TB drive offers (when formatted of course). This is where my “My Documents” folder is stored, amongst my Dads files, old archived files from years ago, and installation ISO’s and packages. I back this drive up 3 times a week rather than daily, as very little data changes on it every day, probably less than a few MB a day in 99% of cases.

Also, M:\ is only using 20GB of the available 465GB the 500GB drive offers. This only contains the backup of the BA server, which is updated hourly and archived to the backup drive. Thinking about it, it hardly warrants a drive on its own really, but the original reason I did this was so I could have “just” the website data backed up into an image and nothing else. This is only possible if it’s the only thing on the drive. The backup software (Norton Ghost) doesn’t allow the backing up of individual folders, just the entire drive. I do find this better however.

To me, this is a waste. Those two drives could easily be combined. I would lose a little bit of the redundancy, but as the merged drive will be mirrored onto a backup drive, that should be enough. I also mirror the images occasionally to other computers like my desktop and to removable media such as DVD’s too so I do have an archive that isn’t very old containing most of the data somewhere in the house.

I have to sacrifice some redundancy if I am to be able to use a huge amount of the remaining free space throughout the drives effectively without combining data that’s unrelated. To do this, I have combined X:\ into drive M:\. This leaves drive X:\ completely empty and usable for other duties. I’ve never liked mixing drives up as I like things to be neat, but in this case I believe I can make an exception as both sets of files have to be backed up and in the end they’re “working data” files, and much like each other. I can live with that

After the merging, and swapping of drive labels and letters, the layout and labelling now looks like this:

C:\ 80GB (boot) ~10GB used
M:\ 500GB (Data) ~45GB used
N:\ 500GB (Backup) ~250GB used
X:\ 1TB (Betas) ~750GB used
Y:\ 1TB (Media) ~910GB used
Z:\ 1TB (Media 2) 0GB used
(see below for a nicer diagram)

The new data drive is using 45GB of 500GB and the old data drive is now completely empty. I have allocated this 1TB drive as a new media drive, as the current one is only about 20GB away from being full, which I will quick quickly fill in the next week or so. I don’t expect I will fill the remainder of the 500GB data drive any time soon, so I have saved buying myself a new drive, at the, what we could say “tiny” expense of some redundancy.

You should never need more than 1 backup of anything, but as the saying goes, you can’t have enough backups, ever. But remember, I do backup to other machines occasionally too, so the slight drop in redundancy on that front doesn’t bother me too much. Normally it would if I didn’t have that extra protection.

I don’t know what anyone will have learnt from this but I just thought I’d share my experiences with you!

A nicer diagram of the drive layout and info from my status page:
Image

pizzaboy192
User avatar
Posts: 2688
Joined: Thu Oct 23, 2008 3:25 am
Location: Earth.
Contact:

Re: The backup predicament… not enough drives!

Post by pizzaboy192 »

That is a really smart way to save some money...
My setup for backups goes as follows:
c: Windows 7 installation (160gb) (28gb free)
d: Media and .iso files (160gb) (29gb free)
e: Complete and utter useless waste of a Molex jack (80gb) (empty)
L: External disk, Windows backup (160gb connected to firewire) (116gb free)

and then I have my fileserver:
c: Windows and Website (40gb) (28gb free)
d: FTP, Torrents, Un-completed media downloads (40gb) (10gb free)

I decided to mirror my website backup across my L: drive on my desktop, and my d: drive on my server... not that there is anything that needs to be saved...
The biggest problem I have is what to do with that 80gb drive that just sits there... I was thinking of placing it back inside my external enclosure, or building it into a new media server... but I just can't bring myself to taking a drive out of my pc... I am thinking an Ubuntu installation or an OSX install may be in order... =)

Thanks for the tips on how to save some dough

Rioter
User avatar
Donator
Posts: 2480
Joined: Sun Nov 09, 2008 12:09 am
Location: Bristol, UK

Re: The backup predicament… not enough drives!

Post by Rioter »

See this is is is... just confusing

No other way of putting it.

I guess with huge amounts of data, it's like ugh where does it all go.
Image

expert01
Posts: 352
Joined: Thu Aug 24, 2006 6:28 pm

Re: The backup predicament… not enough drives!

Post by expert01 »

I got my 8 drive RAID card in last week, so I'll have my storage drives as RAID 6 with a spare hot-swap drive (to automatically rebuild a damaged drive). That's plenty of protection for me. But I just got my replacement Simpletech PRo Duo 1TB in today, and it has two 500GB drives which can be set as RAID 1, so I'll use that to backup the server's OS and probably my desktop computer.

The Distractor

Re: The backup predicament… not enough drives!

Post by The Distractor »

haha, my drive letters are just complicated -

C: - 70GB, windows, apps & some data
D: - 500GB, data & downloads with a few backups of the C: drive
E: - LG DVD burner with lightscribe
F: - PowerISO virtual drive for iso mounting (i upgrade windows by mounting isos, etc)
G: - 400GB external USB drive, where i used to have all downloads+data until i got my box with the drive D: in it. Needless to say, a lot of stuff is here.

All drive sizes are unformatted capacities.
Very unorganised

Hackerpcs
Donator
Posts: 441
Joined: Mon Feb 25, 2008 3:57 am
Location: Greece

Re: The backup predicament… not enough drives!

Post by Hackerpcs »

C: - 500GB, Only Game Installations.
D: - 70GB, Windows XP
E: - 35GB, Windows 7
F: - 1TB, 1080p Movies, Game Installations
G: - 250GB, Music, Videos, Software, Some low-res movies, External.
H: - 250GB, Greek series, RapidShare Downloads, Virtual Machines, External.
I: - LG GGW-H20L, Blu-ray Recorder
J: - 1TB, Beta-Old-RTM Software (A little mirror to Betaarchive, approx. 630GB, and downloading now 23GB)
K: - 200GB, Temp (temporary stored dvds, needs to be burned sometime), Full Metal Jacket 1080p and my favourite team's videos PANATHINAIKOS
L: - Plextor PX-810SA Drive
M: - LG GH-22LP DVD-RAM
N: - Virtual Drive
D:, E: and K: are one 500GB disk. There are 2 Linux partitions, 86GB and 22GB.
In my opinion as organized as possible.

My worst habit in my life: Never back up.

oasiz
User avatar
Donator
Posts: 305
Joined: Sun Jul 20, 2008 12:30 am
Location: Finland
Contact:

Re: The backup predicament… not enough drives!

Post by oasiz »

Yeah, this seperation of drives really annoys a lot when you have to start moving stuff around to make more space.
I have made a bad solution to this with raid0, I don't really care much for redunancy since I have most of the critical stuff mirrored in another arrays.

My current setup:
/dev/sda - Memory stick, (/boot/) temporary for installation purposes.

/dev/sdb + /dev/sdc - 2x 1tb (/dev/md0) mapped as Y: for bulk storage in /home/oasiz/ and U: for /var/www/
/dev/sdd + /dev/sde - 2x 160gb (/dev/md1) mapped as Z: for bulk storage.
/dev/sdf + /dev/sdg - 2x 500gb (/dev/md2) mapped as X: for bulk storage.
/dev/sdh, partitioned to 1gb swap, 128mb /boot/ and rest is for future (you guessed it!) bulk storage.
So that totals a seven drive configuration for the server (plus 2gb stick for temporary boot since I am too lazy to reinstall grub to sdh)

Main system has a louzy 2x 74gb raptor configuration in raid0. Making it the C drive.
Plextor DVD BBQ at D: and floppy keeping A: occupied. So that's C Y X Z K D A U.

Arrays were sure a great joy to set up from commandline by hand. (not knowing the drive locations/sdx markings at all!)

Yeah, OSs should get some options to auto-detect media type and sort them accordingly if one wishes. Also using locations instead of letters, but that is just a dreamworld.

thenico
Posts: 131
Joined: Wed Jun 27, 2007 8:13 pm
Location: France
Contact:

Re: The backup predicament… not enough drives!

Post by thenico »

oasiz, on linux, look /dev/disk/.
You can select a disk (and its partitions) by its uuid, by its hardware path or by its name.

oasiz
User avatar
Donator
Posts: 305
Joined: Sun Jul 20, 2008 12:30 am
Location: Finland
Contact:

Re: The backup predicament… not enough drives!

Post by oasiz »

Wow, cool.

This should help a lot when I have to fiddle with it next time. Thanks
Sure beats checking everything by hand with fdisk!

Post Reply