This whole post comes about thanks to a very special member of the forum, whom without this would not have been possible.
“Viper”, a long-standing member from all the way back in 2008, just a few years after we first opened our doors, has taken it upon himself to single handedly secure the near future BetaArchive repository by donating to us £1,000.
This is a huge sum of money for which we are immensely grateful. It has helped us to secure the future of BetaArchive’s repository in the way of storage. We have struggled to fund buying new hard drives for BetaArchive because of how large we have grown and how expensive large drives are. Not only this, but managing such a large amount of data has become a dangerous affair because it is a lot of data to lose should anything go wrong.
Thanks to this donation (and the luck of receiving it on Amazon Prime day), we have secured 60TB of additional disk space which will go towards upgrading our production drives. Given how good of a deal this Amazon Prime Day deal was, mrpijey was unable to pass up the deal has also invested significant money of his own money (putting him close to a state of bankruptcy
) to secure an additional 60TB of disk space.This gives us a total of 120TB of additional disk space dedicated to BetaArchive’s repository.
What does this mean for BetaArchive’s future?
If used together, 120TB of physical disks gives us the potential to store 405TB of deduplicated data if we maintain the same ratio we have now (3.38).
However, it will be split between production and 2 backups, therefore giving us 135TB of archive space to work with compared to the current 108TB.
The old disks will be repurposed for releases that are being processed and stored ready for release, as well as upgrades to other parts of the infrastructure.
Whilst it doesn’t sound like a huge upgrade when we split it up like this, it dramatically improves our daily workflow. mrpijey was having to juggle data between a total of 1TB of remaining disk space between the drives, and because of this, adding new releases was becoming difficult because deduplication has to occur after the data is written to the drive. With more free space now available it will be a lot easier balancing the data, and with the added physical storage, deduplication rates will improve.
In addition to this we’re also considering building an array out of the disks which will not only improve the overall deduplication rate by performing it across all data rather than just data on each disk, but also adds in hardware redundancy in the form of parity. We will be using ReFS instead of NTFS to protect the data against bit rot and to further improve performance. We will test the feasibility of this first before making a final decision. We have had some disasters when working with RAID5/6 arrays in the past but with much improved hardware and software we’re once again evaluating the storage solution.
We will continue to use Windows as our primary operating system as it has proven to be reliable and give us the flexibility and options we require.
Server Upgrade
In addition to upgrading BetaArchive’s storage, we also want to upgrade our web server. The current server is not struggling, but it is beginning to run out of disk space and this will become a problem with our future plans (more on those later).
For a little more monthly cost (€10 more) we can upgrade the server with double the memory (32GB to 64GB) and double the disk space (1TB to 2TB) as well as move from SATA SSDs to NVMe drives. The processor will also go from an i7-7700 to an i7-8700 giving 50% more performance.
We’re hoping to have this purchased by the end of the month and up and running inside of a week. We’ll keep you up to date on this one as it happens.
Site Updates
Where to begin… We have a number of plans which we hope to see progress with before the end of this year, but you know us, our time estimates are as good as a plumbers estimate…
Database
We plan to add additional data including (but not limited to):
- Artwork
- Disc and track metadata such as hashes, ring code IDs, application IDs etc.
- Tighter integration with other preservation communities and resources (for hash comparison etc.)
- Regional and language info
This is all still in development so the feature list is subject to change during development. You will of course get access to beta features as they’re built and tested.
Website
As part of plans to bring the site more generally up to date we will introduce more information pages to the site.
To make it easier to build and edit pages we’ll build it using a CMS. The kind of information we will include to begin with are fully documented step-by-step guides on how to make software dumps. The guides on the forum will be removed in favour of these new pages.
We also plan to write some basic guides on how to get started with collecting, installing operating systems by explaining how to use virtual machine software, and more.
We also want to make an editorial section of the site where we can post news and updates related to the site and software, but this will come after the above changes.
A Final Thank You
BetaArchive continue to operate as a non-profit website because we don’t feel it would be fair (or legal) to charge for access to software created by others, so we will always rely on your donations to keep us online. We won’t be able to sustain growth of the site otherwise, so we do need your continued help.
Once again we would like to thank Viper for his generous donation which has made all of this possible. But we would also like to thank every single one of you that has donated to BetaArchive since we opened, either through monetary donations or software. Without you we would not be where we are today.
Sincerely,
Andy, mrpijey, and all BA Staff
Thats a lot.
Not that that would ever happen, but if that just somehow happened, what would you do?