BetaArchive Logo
Navigation Home Database Screenshots Gallery Image Uploader Server Info FTP Servers Wiki Forum RSS Feed Rules Please Donate
UP: 35d, 7h, 50m | CPU: 15% | MEM: 5836MB of 12287MB used
{The community for beta collectors}

Post new topic Reply to topic  [ 9 posts ] 
Author Message
 PostPost subject: FTP upgrades        Posted: Mon Mar 06, 2017 12:25 pm 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
7588
It never ends... I've ordered brand new harddrives for BA to replace and merge current ones and I will be (slowly) migrating half of BA FTP to the new drives over the next few weeks. This means a lot of copying and even more deduplicating which futher means that the performance may slow down at times during heavy download sessions as I will set the system to prioritize local I/O rather than remote. On top of this I also need to update the backup media and make sure the backup keeps up with the changes. There won't be any changes in the FTP folder layout so the harddrive shift shouldn't be noticable for any of you. But once it's done we'll get newer, quicker and larger harddrives to hold all the goodies we have!

Will update with more info once things gets moving.

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Tools: Alcohol120% (Portable)
Listings: BetaArchive Database (beta)
Channels: Discord :: Twitter


Top  Profile  WWW
 PostPost subject: Re: FTP upgrades        Posted: Tue Mar 07, 2017 1:32 am 
Reply with quote
FTP Access
User avatar
Offline

Joined
Tue Jul 08, 2014 9:43 pm

Posts
418

Location
DLL Hell, United States

Favourite OS
Windows 2000 SP4 and XP SP3
Frank Zappa wrote:
In the room where the giant BA server works and the torture never stops

Joking aside, take your time.

_________________
Do Not Make Illegal Copies Of This Signature.
YouTube | Twitter | BA Wiki | BetaWiki


Top  Profile  WWW
 PostPost subject: Re: FTP upgrades        Posted: Tue Mar 07, 2017 4:54 am 
Reply with quote
Donator
Offline

Joined
Sat Sep 09, 2006 6:43 am

Posts
746

Favourite OS
Win10/Debian Linux
I gotta ask, what is the technical makeup of the data store? To store as much data as the archive contains, you've got to have a really nice setup in place. One day, I'll probably have to setup such an archive; local storage can only go so far.

_________________
Need disks scanned in the USA? I have a Kryoflux, and am willing to help get your disks archived! I also offer xbox and xbox 360 repair and modding services. PM me for details!


Top  Profile  WWW
 PostPost subject: Re: FTP upgrades        Posted: Tue Mar 07, 2017 8:43 am 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
7588
Windows OS wrote:
Frank Zappa wrote:
In the room where the giant BA server works and the torture never stops

Joking aside, take your time.

You got that right, it is torture... for me :)

jimmsta wrote:
I gotta ask, what is the technical makeup of the data store? To store as much data as the archive contains, you've got to have a really nice setup in place. One day, I'll probably have to setup such an archive; local storage can only go so far.


I did provide some info during the server swapout (topic here), but if you want to know any more specific details about it all just ask in that thread and I'll see if I can answer them... I am going to update that topic soon with more pics as I've gotten a new rack cabinet to hold it all as well as new drives etc. Was thinking about posting some screenshots of the various drive setups etc (if anyone is interested that is).

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Tools: Alcohol120% (Portable)
Listings: BetaArchive Database (beta)
Channels: Discord :: Twitter


Top  Profile  WWW
 PostPost subject: Re: FTP upgrades        Posted: Tue Mar 07, 2017 9:37 pm 
Reply with quote
FTP Access
User avatar
Offline

Joined
Tue Jul 05, 2016 6:57 am

Posts
138
I am interested in your setup, mrpijey. Just a bit curious is all.


Top  Profile
 PostPost subject: Re: FTP upgrades        Posted: Tue Mar 07, 2017 10:09 pm 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
7588
I will post some updates once everything is in place. I've installed the harddrives and they are being prepared now, tomorrow begins The Big Copy of all the data. I do not look forward to this...

But even if the log isn't updated during this copy (the reason will be clear later) I am still adding member uploads to the FTP, all of it will be reported in the next update log.

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Tools: Alcohol120% (Portable)
Listings: BetaArchive Database (beta)
Channels: Discord :: Twitter


Top  Profile  WWW
 PostPost subject: Re: FTP upgrades        Posted: Wed Mar 08, 2017 4:32 am 
Reply with quote
FTP Access
User avatar
Offline

Joined
Tue Jul 08, 2014 9:43 pm

Posts
418

Location
DLL Hell, United States

Favourite OS
Windows 2000 SP4 and XP SP3
mrpijey wrote:
I will post some updates once everything is in place. I've installed the harddrives and they are being prepared now, tomorrow begins The Big Copy of all the data. I do not look forward to this...

Well, I certainly wish you luck, man. Hopefully, everything goes without a hitch. Good luck ;)

Offtopic Comment
mrpijey wrote:
Windows OS wrote:
Frank Zappa wrote:
In the room where the giant BA server works and the torture never stops

Joking aside, take your time.

You got that right, it is torture... for me :)

I wonder if you got the joke. If you didn't, that quote is a modified lyric coming from the Frank Zappa song "The Torture Never Stops", an excellent blues rock song that I recommend to anyone who likes 70s classic rock

_________________
Do Not Make Illegal Copies Of This Signature.
YouTube | Twitter | BA Wiki | BetaWiki


Top  Profile  WWW
 PostPost subject: Re: FTP upgrades        Posted: Thu Mar 09, 2017 1:21 am 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
7588
Well, seems things are not as smooth as possible, the copy speed is much lower than expected. To make this a lot quicker I will have to take the FTP archive offline and do an in-copy of the files, which means that I will bring the new drives online as primary BA drives and use two other servers to copy from at the same time, thus speeding up the copy. This means that parts of the archive will be offline until everything has been copied over. It will appear as the FTP are missing sections and files, but these will be added again as the copy progresses...

During this night I will copy the data as usual, but tomorrow I will do the switch. It's more important to copy over the data onto the new drives as quickly as possible, and at the same time being able to add new fresh builds inbetween as they are being released on the net. Speed is essential here as I need to copy all the data from the old drives and also do the backup update at the same time, and to have multiple machines copy in the data onto the new drives at the same time greatly increases speed, and at the same time being able to alternate between copy and dedup between the drives without any interruption.

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Tools: Alcohol120% (Portable)
Listings: BetaArchive Database (beta)
Channels: Discord :: Twitter


Top  Profile  WWW
 PostPost subject: Re: FTP upgrades        Posted: Wed Mar 15, 2017 12:30 pm 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
7588
Good news, all the merging from the old drives is now complete. I will add a few more new things to the FTP and then do a major update log as I've added a lot of new stuff to the FTP. But nothing is missing from the "old" drives and everything is in place. I will continue to do backups in the background but it won't affect any of the contents.

We are now sitting on 2x8TB drives + 2x4TB drives (=24TB physical storage), all backed up to 3x8TB. Further ahead I will replace the 4TB drives as well, but for now it will do.

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Tools: Alcohol120% (Portable)
Listings: BetaArchive Database (beta)
Channels: Discord :: Twitter


Top  Profile  WWW
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 9 posts ] 




Who is online

Users browsing this forum: No registered users and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  

All views expressed in these forums are those of the author and do not necessarily represent the views of the BetaArchive site owner.

Powered by phpBB® Forum Software © phpBB Group

Copyright © 2006-2019

 

Sitemap | XML | RSS