BetaArchive Logo
Navigation Home Database Screenshots Gallery Image Uploader Server Info FTP Servers Wiki Forum RSS Feed Rules Please Donate
UP: 76d, 2h, 48m | CPU: 15% | MEM: 6568MB of 12287MB used
{The community for beta collectors}

Post new topic Reply to topic  [ 18 posts ] 
Author Message
 PostPost subject: Backups... Never easy are they?        Posted: Fri Jun 19, 2009 4:59 pm 
Reply with quote
Administrator
User avatar
Offline

Joined
Fri Aug 18, 2006 11:47 am

Posts
12547

Location
Merseyside, United Kingdom

Favourite OS
Microsoft Windows 7 Ultimate x64
Hello all,

Well second blog post in 2 days. Not bad... Well, it wouldn't be if it wasn't me ranting on about the problems I've been having. This time its backups.

The problem this time is Norton Ghost 9. Its doing my head in. Its slow, buggy and very resource intensive. So much infact, that it uses a full CPU core when restoring files, and takes ~6 hours to restore 15GB of data from one of the archive images. I find this ridiculous and I urgently need to find a new piece of software to do my backups.

My choices are limited though, since I need the Volume Shadow Copy Service for my backups to work properly. The reason being that a lot of files are "locked" by the operating system when they are in use, and can't be copied with conventional methods. That's where VSS comes in. It allows the copying of locked files so you can make backups of the entire drive without dis-mounting it. Ghost does this, but the version I currently have doesn't like restoring files very efficiently.

I have several choices that have cropped up so far, and even they are limited...

Norton Ghost 14 - A newer version of what I currently have. I just hope the coding is more optimised in this version.
Acronis True Image - Pretty much a Norton Ghost copy but has more features than the current Ghost version I'm using.

Either way as long as it works the same way as my current backups do, and restores quicker, I'll be happy. The hard part is getting hold of a copy of either piece of software, since as you know they're quite costly and my searches so far have all drawn a blank or trojan infested archives. I'll keep looking ;)

For now, I am having to deal with the slow restoring of Norton Ghost 9, since its the only software that can read the files it created...

_________________
Image

BetaArchive Discord: https://discord.gg/epK3r6A


Top  Profile  WWW
 PostPost subject: Re: Backups... Never easy are they?        Posted: Fri Jun 19, 2009 7:49 pm 
Reply with quote
Staff
User avatar
Offline

Joined
Thu Nov 30, 2006 6:10 pm

Posts
2665

Location
Land of The Tea.

Favourite OS
Windows 10 Pro x64
Norton Ghost is good, and fastish on my C2Q, only took about 4 hours to compress and send over network 108GB

_________________
Image
| Personal site - Social links - PC setup | TF2 Game Servers |
Liking traps isn't gay, its not gay if it looks like a girl
I-i-it's n-not as if I wanted to ban you or anything. B-baka. (「・ω・)「


Top  Profile  WWW
 PostPost subject: Re: Backups... Never easy are they?        Posted: Fri Jun 19, 2009 8:51 pm 
Reply with quote
Donator
User avatar
Offline

Joined
Sun Jul 15, 2007 5:29 pm

Posts
1009

Location
Sweden

Favourite OS
OSX
Andy,

If you need backup, you can get some software from my new company (www.sabas.dk) (sorry, only in danish), its a piece of software that can do both online and offline backups.

I'm back sunday, lets have a talk there.

_________________
Regards,
Leon Bollerup - BA VIP Member

Run OSX on a pc - Ahsay backup monitoring - Online Backup


Top  Profile  WWW  ICQ
 PostPost subject: Re: Backups... Never easy are they?        Posted: Sat Jun 20, 2009 5:25 pm 
Reply with quote
Administrator
User avatar
Offline

Joined
Fri Aug 18, 2006 11:47 am

Posts
12547

Location
Merseyside, United Kingdom

Favourite OS
Microsoft Windows 7 Ultimate x64
Well, I've just installed Norton Ghost 14 in place of Norton Ghost 9 and I can say it is indeed much faster at restoring files, so that has made me a very happy person! The only downside is it is still single threaded so its not making full use of the resources available so its not as fast as it really could be had it made use of both cores. And, like all programs, its speed at copying lots of small files is indeed slow, just like FTP and Windows file copying, but that was to be expected. However, it is faster and that's what I was after.

What took v9 1 hour to get 1/3 of the way through, v14 has managed to totally complete in around 20-25 minutes. :) That's almost 1.5GB of data by the way, so its pretty good going.

Another advantage of v14 is the ability to backup just files/folders and not just a drive, which again has made me a very very happy person since my backups can be split without having multiple partitions and/or drives.

_________________
Image

BetaArchive Discord: https://discord.gg/epK3r6A


Top  Profile  WWW
 PostPost subject: Re: Backups... Never easy are they?        Posted: Sat Jun 20, 2009 7:48 pm 
Reply with quote
Administrator
User avatar
Offline

Joined
Fri Aug 18, 2006 11:47 am

Posts
12547

Location
Merseyside, United Kingdom

Favourite OS
Microsoft Windows 7 Ultimate x64
Right, an addition to this.

It seems although its faster, its still way too slow when it comes to smaller files. Put it this way Windows Explorer is faster.

However that's not the only problem with my backups. Because some files are locked while they are in use (such as MySQL and the E-mail server) its impossible to copy them without VSS and there is no easy way to do this without specialised software that I know of. What I need to be able to do is:

Code:
# copy MySQL directory using VSS > /backup
# zip /backup > /backup/mysql_backup.zip
# copy /backup/mysql_backup.zip > Home server
# archive mysql_backup.zip hourly and keep 7 days worth

Repeat for all the other directories on the server that need backing up. Incremental backups are of course better but at a push this would work fine.

I'm off to brain storm. Any ideas people?

_________________
Image

BetaArchive Discord: https://discord.gg/epK3r6A


Top  Profile  WWW
 PostPost subject: Re: Backups... Never easy are they?        Posted: Sun Jun 21, 2009 7:07 pm 
Reply with quote
Administrator
User avatar
Offline

Joined
Fri Aug 18, 2006 11:47 am

Posts
12547

Location
Merseyside, United Kingdom

Favourite OS
Microsoft Windows 7 Ultimate x64
More updates on this problem...

Last night I tried SyncBackSE's backup facility, to find it allowed something called "Fast Backups", which is very similar to what rSync does, except it doesn't allow incremental backups of the data in a file, only files themselves (E.G. It won't just add onto a large file, it copies the whole thing, but it doesn't have to do every file in the backup, just that single file).

Having found this, I did a variety of tests...

Code:
# Samba Network Share via Hamachi
# FTP


Samba Network Share via Hamachi
It sounds ingenious doesn't it? Well, it is in many ways, but its not ideal for many reasons too.

Some of you know Hamachi, some of you don't. For those who don't, its virtual VPN software that runs on various platforms, and allows you to do what a VPN was designed to do - access your local network away from home. It works perfectly and does what it says on the tin, but Samba is simply too slow over Hamachi to be useful in this application. It changes directories much too slowly (0.5-1 directory a second) and when you have several thousand folders to switch between when scanning, its very very slow.

File transfer speeds are pretty good considering the sense it is being used. I managed around 5.5Mbps - much much better than rSync's 0.6Mbps. I was very happy with those speeds and gave it more testing.

Upon these further tests, I found that it was failing to write to several files that were locked on the system. Although SyncBackSE uses VSS it only uses it locally, and not on the other end as well, so it failed to write some files, and this was going to be a problem. Time to test FTP.

FTP
FTP is of course fairly simple so I won't bother explaining it.

Changing directories with FTP was a little faster, managing around 3-4 directories a second. Still slow, but much better than Samba and definitely a lot more bearable.

File transfer speeds were pretty much the same, around 5.5Mbps so thats good as well.

Now, the file writing - it works with FTP! Strange though it may seem, FTP was able to write to the locked files that Samba was not. I can't explain it, but I'm sure everyone has experienced the "File in use" error when trying to delete a file with Windows Explorer before, even though you know for sure its not being used. I put it down to this and just accepted FTP was a better choice for this application.


So, SyncBackSE is now my new backup software. It connects to my home server via FTP hourly for database/e-mail backups and every 12 hours for the rest of the files. Norton Ghost 14 has now been put in place of Norton Ghost 9 for creating the image files and that is working well too. SyncBackSE also allows me to select the files and/or folders I want and not just a base folder. I have full control over what I want to backup, unlike rSync.

So there you have it folks, 2 days worth of trouble and I have a new backup solution. Lets see how it holds up. I'll get back to you with any problems I have should it come down to it!

_________________
Image

BetaArchive Discord: https://discord.gg/epK3r6A


Top  Profile  WWW
 PostPost subject: Re: Backups... Never easy are they?        Posted: Sun Jun 21, 2009 7:12 pm 
Reply with quote
FTP Access
User avatar
Offline

Joined
Sat Aug 19, 2006 7:45 pm

Posts
550
I thought Symantec Ghost Solution Suite (the corporate version of Ghost) was a lot better than the consumer version, being able to backup to full GHO images and not the horrible PowerQuest formats.

_________________
I am here in the shadows


Top  Profile
 PostPost subject: Re: Backups... Never easy are they?        Posted: Sun Jun 21, 2009 7:47 pm 
Reply with quote
Staff
User avatar
Offline

Joined
Thu Nov 30, 2006 6:10 pm

Posts
2665

Location
Land of The Tea.

Favourite OS
Windows 10 Pro x64
Syncbackup SE is good and fast even oner FTP. I use.

_________________
Image
| Personal site - Social links - PC setup | TF2 Game Servers |
Liking traps isn't gay, its not gay if it looks like a girl
I-i-it's n-not as if I wanted to ban you or anything. B-baka. (「・ω・)「


Top  Profile  WWW
 PostPost subject: Re: Backups... Never easy are they?        Posted: Sun Jun 21, 2009 7:50 pm 
Reply with quote
Administrator
User avatar
Offline

Joined
Fri Aug 18, 2006 11:47 am

Posts
12547

Location
Merseyside, United Kingdom

Favourite OS
Microsoft Windows 7 Ultimate x64
TheWahbinator wrote:
I thought Symantec Ghost Solution Suite (the corporate version of Ghost) was a lot better than the consumer version, being able to backup to full GHO images and not the horrible PowerQuest formats.


I honestly can't say since I've never used it, but I do know .GHO support was dropped in Ghost 14...

_________________
Image

BetaArchive Discord: https://discord.gg/epK3r6A


Top  Profile  WWW
 PostPost subject: Re: Backups... Never easy are they?        Posted: Sun Jun 21, 2009 8:56 pm 
Reply with quote
Donator
Offline

Joined
Sat Oct 04, 2008 5:43 pm

Posts
1237

Location
Milky Way Galaxy

Favourite OS
Windows Server 2012 Dtc
I have an internal version of Norton Ghost (Symantec Internal) its quite old, but it runs on Windows PE, and it has some cool Symantec Internal Tools, and backing up is so easy - It has compression as MAX and it made my 1 TB drive into 890GB! :) :D

_________________
See my profile for my website link.


Top  Profile  WWW
 PostPost subject: Re: Backups... Never easy are they?        Posted: Sun Jun 21, 2009 10:59 pm 
Reply with quote
Donator
User avatar
Offline

Joined
Mon Sep 04, 2006 1:06 pm

Posts
1004

Location
USA
motherboardlove wrote:
I have an internal version of Norton Ghost (Symantec Internal) its quite old, but it runs on Windows PE, and it has some cool Symantec Internal Tools, and backing up is so easy - It has compression as MAX and it made my 1 TB drive into 890GB! :) :D


Any chance of an upload?

_________________
Old-Computer
BetaArchive VIP | Ex-OSBA Member


Top  Profile
 PostPost subject: Re: Backups... Never easy are they?        Posted: Mon Jun 22, 2009 9:05 pm 
Reply with quote
Staff
User avatar
Offline

Joined
Thu Nov 30, 2006 6:10 pm

Posts
2665

Location
Land of The Tea.

Favourite OS
Windows 10 Pro x64
Jeff wrote:
motherboardlove wrote:
I have an internal version of Norton Ghost (Symantec Internal) its quite old, but it runs on Windows PE, and it has some cool Symantec Internal Tools, and backing up is so easy - It has compression as MAX and it made my 1 TB drive into 890GB! :) :D


Any chance of an upload?

2'nd

_________________
Image
| Personal site - Social links - PC setup | TF2 Game Servers |
Liking traps isn't gay, its not gay if it looks like a girl
I-i-it's n-not as if I wanted to ban you or anything. B-baka. (「・ω・)「


Top  Profile  WWW
 PostPost subject: Re: Backups... Never easy are they?        Posted: Mon Jun 22, 2009 10:49 pm 
Reply with quote
Donator
Offline

Joined
Sat Oct 04, 2008 5:43 pm

Posts
1237

Location
Milky Way Galaxy

Favourite OS
Windows Server 2012 Dtc
Toshua123 wrote:
Jeff wrote:
motherboardlove wrote:
I have an internal version of Norton Ghost (Symantec Internal) its quite old, but it runs on Windows PE, and it has some cool Symantec Internal Tools, and backing up is so easy - It has compression as MAX and it made my 1 TB drive into 890GB! :) :D


Any chance of an upload?

2'nd


Well, the disc is here:

Image

Im no sure about ULing it - it is for Symantec Employees only, and for internal use only, but Ill think about it....

_________________
See my profile for my website link.


Top  Profile  WWW
 PostPost subject: Re: Backups... Never easy are they?        Posted: Tue Jul 07, 2009 2:14 pm 
Reply with quote
Donator
User avatar
Offline

Joined
Sat Jan 26, 2008 11:57 pm

Posts
900

Location
Florida

Favourite OS
Windows 7
may i ask how u got your hands on it :P

_________________
Image


Top  Profile  YIM
 PostPost subject: Re: Backups... Never easy are they?        Posted: Wed Jul 08, 2009 5:38 am 
Reply with quote
Donator
Offline

Joined
Fri Oct 26, 2007 5:12 pm

Posts
2461
angelwolf71885 wrote:
may i ask how u got your hands on it :P


Clearly torrented.


Top  Profile
 PostPost subject: Re: Backups... Never easy are they?        Posted: Sat Jul 11, 2009 4:52 pm 
Reply with quote
Staff
User avatar
Offline

Joined
Thu Oct 11, 2007 9:13 pm

Posts
2023

Location
United States

Favourite OS
MacOS 9.2.2
happy dude: agreed

backups are a pain in the crap sometimes. I backed up my entire server, and a lot of the data spontaneously disappeared. :|

_________________
James *~*~* BA Moderator | Alternate History writer


Top  Profile
 PostPost subject: Re: Backups... Never easy are they?        Posted: Sat Jul 11, 2009 5:07 pm 
Reply with quote
Administrator
User avatar
Offline

Joined
Fri Aug 18, 2006 11:47 am

Posts
12547

Location
Merseyside, United Kingdom

Favourite OS
Microsoft Windows 7 Ultimate x64
I just changed the way I do my backups for good.

I wrote my own script using PHP, XCOPY and RD that would perform the backups the way I want it to. It basically works like this:

# Load a text file with the backup number in
# Delete the old backup using RD (backup number - 2)
# Create the new backup using XCOPY (backup number + 1)
# E-Mail the result to my backup e-mail address

Here is the code for anyone interested. Create a file called backup-number.txt and enter a number into it. Remember to edit the locations and the e-mail address also.
Code:
<pre>
<?php

ob_start();

$name = "Test Backup";
$from = "E:\\test";   
$to = "N:\\test_backup";

$fp = fopen("backup-num.txt", "r");
$num = fread($fp, "100000");
fclose($fp);

$numold = $num-2;
$numnew = $num+1;

echo "
Backup in progress...\n
Backup number: ".($num+1)."\n
";

$del = 'rd '.$to.'\\'.$numold.'\\ /S /Q';
echo "Deleting: ".$del."\n\n";
$delete = passthru($del, $erra);
echo $delete;

$run = 'xcopy "'.$from.'\\*.*" "'.$to.'\\'.$numnew.'" /C /E /I /Y /H /R';
echo "Command run: ".$run."\n\n";
$create = passthru($run, $errb);
echo $create;

$success = 0;

if($erra == 0){
 echo "\n\nThe delete was successful.";
 $success = $success + 1;
}
else{
 echo "\n\nThe delete failed.";
 $success = $success - 1;
}

if($errb == 0){
 echo "\n\nThe backup was successful.";
 $success = $success + 1;
}
else{
 echo "\n\nThe backup failed.";
 $success = $success - 1;
}

$num++;
$fp = fopen("backup-num.txt", "w+");
fwrite($fp, $num);
fclose($fp);

ini_set("SMTP","smtp.yourdomain.com" );
ini_set('sendmail_from', 'backup@yourdomain.com');

$to      = 'you@yourdomain.com';
$headers = 'From: you@yourdomain.com';

mail($to, $subject, $message, $headers);

$dump = ob_get_contents();
ob_end_clean();

if($success == 2){
 $subject = "[Success] ".$name;
 $message = $dump;
 mail($to, $subject, $message, $headers);
}
else{
 $subject = "[Failed] ".$name;
 $message = $dump;
 mail($to, $subject, $message, $headers);
}

echo $dump;

?>
</pre>


You have to run the script as a cron job otherwise if you have safe_mode enabled etc it will cause errors, since passthru() is a blocked command under safe_mode. You can do this by having scheduled tasks run the following bat file code (but remember to make a separate copy of PHP for cron use, so you can use another php.ini file (or use the command line switch for choosing another php.ini)):

Code:
c:\php\php.exe "c:\htdocs\backups\backup.php"


The script as it is now will keep 3 copies of a backup before deleting the oldest. To increase this number, change "$numold = $num-2;" to, for example "$numold = $num-24;" to keep 24 copies before deleting the oldest. Remember that you'll need enough space in your backup directory to allow this.

I run 4 backups on my server:
# htdocs (local htdocs data) every 24 hours at 3am
# Data (my personal documents) every 24 hours at 3.30am
# www_data (BA data) every 24 hours at 4am
# DB (MySQL data) every 3 hours starting at 1.45am daily

The backups from the BA server to my house is run via FTP and they run every 12 hours for main data and every 3 hours for database data. The database data backup from the BA server is 1hr 45mins behind the local backup task since that gives it time to finish under the most extraordinary circumstances of it being very slow (normally it takes under 10 minutes to complete).

It sounds complicated, but its not, and its a hell of a lot better than Norton Ghost, apart from the fact that it doesn't support incremental backups. What it does ensure it that you can quickly restore data if necessary without unpacking it from a huge image file whose backend coding is very inefficient! However, you should keep your anti-virus up to date, since it can easily infect the backups too. I now choose to keep regular backups on CD as well as hard disk media.

Give it a try if you can, its a very good solution. :)

_________________
Image

BetaArchive Discord: https://discord.gg/epK3r6A


Top  Profile  WWW
 PostPost subject: Re: Backups... Never easy are they?        Posted: Thu Aug 20, 2009 12:27 pm 
Reply with quote
FTP Access
User avatar
Offline

Joined
Tue Aug 18, 2009 6:42 pm

Posts
22

Location
NE Indiana, USA

Favourite OS
Hera1O9
Andy wrote:
...
The problem this time is Norton Ghost 9. Its doing my head in. Its slow, buggy and very resource intensive. So much infact, that it uses a full CPU core when restoring files, and takes ~6 hours to restore 15GB of data from one of the archive images. I find this ridiculous and I urgently need to find a new piece of software to do my backups.
...


You have my sympathy.

I recently used PING (PING Is Not Ghost). I've had a lot of problems with it. I was helping out a neighbor who was sending his laptop in to Dell and wanted a backup. Dell sent him a new laptop, and none of his old files. "Not a problem. That's what the backup was for.", I thought.

Several days and several restore attempts later (I was using a slow USB drive), I had gotten back most of the guy's documents, most of his music files, but lost over half of the pictures.


Top  Profile  WWW
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 18 posts ] 




Who is online

Users browsing this forum: No registered users and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  

All views expressed in these forums are those of the author and do not necessarily represent the views of the BetaArchive site owner.

Powered by phpBB® Forum Software © phpBB Group

Copyright © 2006-2019

 

Sitemap | XML | RSS