Coming soon(tm)... HTTP download system!

[Andy]

Yes, you read the topic title correctly! We are eventually going to move to a HTTP download system, and I am currently in the process of building it.

Thanks to user mattarnster, we now have a way to authenticate HTTP downloads against a database so we're looking to build functionality directly into the BetaArchive Database to make it possible to download straight from it and get rid of the FTP system (except for uploading releases).

Here is a very early alpha sneak preview of what I have been working on the past few weeks...





So how will it work?

Simply, if you're logged into the forum you'll be able to download from the database directly.

Clicking download will generate a token which is unique to your download, and will ensure that only you can use it by locking it to your current IP. The token will last for a pre-determined amount of time before expiring, so you can set up the download in your preferred download manager. It will support resuming and multiple threads (hurrah!).

When you start a download, its size will be deducted from your daily quota immediately, but since we have no way of tracking if you complete the download you'll be able to download the same file for as long as the token is valid. This allows you to resume broken downloads and use multiple connections from the same download token.

You'll be able to see your existing download tokens in the manage section, where you'll be able to delete tokens which have not been used, or forcibly invalidate those which have.

There is a section where you'll be able to copy the list of tokens in text format, or any other format that I can code into it depending on what the common download managers accept (please suggest if you'd like to see this).

Remember, this is in early alpha!

I must stress this because this system is brand new, in the making, and is not even ready for testing yet. It will take some time to get things smoothed out, but so far everything is going well. A lot of redesign is likely, and I will probably have to recode things several times to get it how I want it too. I hope to get this completed within a few months maximum.

Although I haven't given you much to go on, I would welcome your early thoughts, opinions, questions, suggestions, etc, etc. so that I can make this download system as best I can for the community, because lord knows you've been demanding it for a long time. I want to get this right!

[anonymous74]

This new system looks really cool! FTP works fine, but it is just a tad bit clunky, and it would save a minute or so of time to be able to search directly for the file I want to download. Thank you for the continued development of BA, we all appreciate it. Looking forward to its release!

I don't have much to offer on the technical side of things, but the system sounds fine from what you've said, as long as it isn't any slower than FTP I'd imagine it would work rather well.

[Blue Horizon]

This would certainly be a nice alternative to WinWorld, especially since they're not the best with sorting out ISOs of builds. Nice to see you're embracing freely available downloads with no contribution rules. It'll be interesting to see how this will affect the landscape of contributions in the future. Keep up the good work!

[anonymous74]

This would certainly be a nice alternative to WinWorld, especially since they're not the best with sorting out ISOs of builds. Nice to see you're embracing freely available downloads with no contribution rules. It'll be interesting to see how this will affect the landscape of contributions in the future. Keep up the good work!

Simply, if you're logged into the forum you'll be able to download from the database directly.

Doesn't look like this would be public downloads.

[JustZisGuy]

Does this mean all of the files in a release will be merged in to one archive for download as well as downloadable individually - like what archive.org does? I can't quite tell what the intent is from the screen shots.

I would assume this system would support http resume?

Would it actually have to replace the FTP system, or could it just augment it? I am aware the FTP server software currently in use has issues.

The nature of the BA archive is a bit different than most. Here it seems a more common need would be to mark multiple files for download, start the download, and step away while all files download.

In my curmudgeony opinion, neither web browsers nor FTP are suitable for managing "modern" sized huge sets of files. I wonder if there is any way to tailor a SCP server for BA's needs? No idea if that would be a practical thing to do.

[Blue Horizon]

Doesn't look like this would be public downloads.

I know, that's why I said it's a nice alternative rather than a flat-out replacement. WinWorld still wins out in the regard no account is required, and so does other sites like Old-Dos and of course Internet Archive. Though it is nice that a collection of such size is at least being put out into the wild in some capacity, even if it is required to log in. This does also make it easier for users to download stuff en-masse and put it somewhere else, even if only a little bit at a time because of the quota limitations.

[anonymous74]

@Blue Horizon
I believe Andy meant accounts with FTP Access. I could be wrong but I would imagine if the archive was being opened to that degree, it would be announced with more fan-fare. Andy will have to confirm.

[SamCool939]

Nice! And for regular members (like me), how much is the download quota limit available?

[PivotGuy246]

Definitely looking forward to this.

I do have one question about the new system, though: are download managers (such as FDM) compatible with this? I would definitely want to reap faster download speeds rather than having to endure slower speeds as a result of Tim Kosse's reluctancy towards implementing multithreaded downloads in FileZilla.

[mrpijey]

This would certainly be a nice alternative to WinWorld, especially since they're not the best with sorting out ISOs of builds. Nice to see you're embracing freely available downloads with no contribution rules. It'll be interesting to see how this will affect the landscape of contributions in the future. Keep up the good work!

We will still require the contribution rules before you get access to the download system. The HTTP system simply replaces the FTP for daily downloads, it doesn't replace the contribution requirements for new members or quota. That hasn't changed.

Nice! And for regular members (like me), how much is the download quota limit available?

Quotas for the time being will be the same. We won't have any more bandwidth just because we move to a http system so we're still limited in what we can provide.

[...]Though it is nice that a collection of such size is at least being put out into the wild in some capacity, even if it is required to log in. This does also make it easier for users to download stuff en-masse and put it somewhere else, even if only a little bit at a time because of the quota limitations.

We're not interested in allowing you to leech the site dry and put it elsewhere, none of those restrictions has changed nor will change. We're on limited capacity throughout the infrastructure and because we're moving to a HTTP system doesn't mean we've suddenly moved to unlimited bandwidth and capacity...

Definitely looking forward to this.

I do have one question about the new system, though: are download managers (such as FDM) compatible with this? I would definitely want to reap faster download speeds rather than having to endure slower speeds as a result of Tim Kosse's reluctancy towards implementing multithreaded downloads in FileZilla.

From initial tests download managers seem to work with this. But it may not guarantee you get any faster speeds, only that you will be able to resume downloads and queue them up. We're still limited on bandwidth and resources, that hasn't changed.

The HTTP system is simply a replacement for the clunky FTP system, to move to a system where you can use your download manager of your choice for easier downloads, without the need to install and configure an FTP client to download something. To upload you will need an FTP client yes, but it can then be replaced and updated to make it easier as well and only used for uploads. The HTTP system will allow you to find what you want in the database and immediately be able to download it, instead of having to start a different client and manually look it up.

[Blue Horizon]

We're not interested in allowing you to leech the site dry and put it elsewhere, none of those restrictions has changed nor will change. We're on limited capacity throughout the infrastructure and because we're moving to a HTTP system doesn't mean we've suddenly moved to unlimited bandwidth and capacity...

I never said I was going to leech off the servers, just that it would increase the likelihood of others maxing out their quotas using the added benefit of having multiple download instances with their download manager (which would vary depending on the traffic the servers get), wait for their quota to reset, then rinse and repeat. Would you have or are planning something to put in place to try and circumvent that by chance?

[Andy]

We would continue to use our discretion for users who abuse the system just as we do with the FTP. We have a fair usage policy in place covering this and it won't change.

[mrpijey]

We're not interested in allowing you to leech the site dry and put it elsewhere, none of those restrictions has changed nor will change. We're on limited capacity throughout the infrastructure and because we're moving to a HTTP system doesn't mean we've suddenly moved to unlimited bandwidth and capacity...

I never said I was going to leech off the servers, just that it would increase the likelihood of others maxing out their quotas using the added benefit of having multiple download instances with their download manager (which would vary depending on the traffic the servers get), wait for their quota to reset, then rinse and repeat. Would you have or are planning something to put in place to try and circumvent that by chance?

Well it wouldn't be any different from today with people queing up tons of downloads in the FTP client and then wait for the quota to refresh when it's depleted. And we will still be able to monitor usage like we do now. So the risk of exploiting it is no more or less than before. The only thing that could happen with this HTTP system is that the downloads will be slightly faster, but that only means they use up their quota faster, which in turn means they will release their bandwidth usage to others faster as well.

[Darkstar]

I, for one, will be sad to see the FTP interface go away. It always felt "natural" for a site like BA, which focuses on old software, to also use a "retro" download protocol

But progress cannot be resisted I guess

[Andy]

Unfortunately it's very unstable these days and we spend far more time keeping it working than we should have to. Plus, the software we use has been unsupported for about a decade now and it's starting to show with SSL incompatibility, stability issues, hacks needed to keep it running on modern operating systems, etc. It's about time it was retired.

[anonymous74]

Will the “FTP Access” rank have its name changed to complement the new download system?

[Andy]

Yeah we can probably change it.

[mibrab]

Since some releases are larger than the current quota, how will these be handled?

[Andy]

That's still to be decided exactly how it will work, but most likely we'll allow you to go over your quota and download it providing you have at least a certain quota percentage available, but I don't know for sure yet. But it is a very good question that needs a good amount of thought to handle properly.

[yourepicfailure]

So I see a potential issue: hacks.

Using HTTPS for downloads, and given history of people hacking this site, it seems quite obvious to me that someone could in fact "break in" and proceed to mirror the contents easily with no regard for bandwidth limitations.
Granted the current ftp situation isn't in any different circumstance.
How are you ensuring that the chances of this are low?

[Andy]

First off I don't see how using HTTPS is going to increase the likelihood of us being hacked, and it's not like we have been hacked before either. Minor exploits through the forum perhaps, but never hacked.

Secondly, it will not be possible to download any file unless the download is authenticated against a token. If you don't authenticate successfully against a valid token, it rejects your connection.

We intend to go through a significant testing period where we get trusted people to try and do things to the system that are not intended, so any issues can be ironed out before a wider release. However, since the system uses the forum's authentication system which we also use for the FTP, I don't currently see any opportunity for this to be exploited.

[Andy]

What's everyone's thoughts on a rolling quota system? It would work like this:

You start a download, its quota is deducted. The quota used will return after a fixed time determined by the file size.

For example:

- A 72GB file would return 72GB of quota 36 hours after it was first downloaded.
- A 24GB file would return 24GB of quota 12 hours after it was first downloaded.
- A 2GB file would return 2GB of quota 1 hour after it was first downloaded.
- and so on...

This will allow you to download large files but wait longer so your daily average can't exceed 50GB, and is the easiest and most flexible way for quotas to work to benefit you as a user.

[XylonJack]

In the future, are you considering supporting online preview of artwork?

[mrpijey]

Yes the database view will be enhanced with artwork, but that's a separate project from this.

[x010]

What's everyone's thoughts on a rolling quota system? It would work like this:

You start a download, its quota is deducted. The quota used will return after a fixed time determined by the file size.

For example:

- A 72GB file would return 72GB of quota 36 hours after it was first downloaded.
- A 24GB file would return 24GB of quota 12 hours after it was first downloaded.
- A 2GB file would return 2GB of quota 1 hour after it was first downloaded.
- and so on...

This will allow you to download large files but wait longer so your daily average can't exceed 50GB, and is the easiest and most flexible way for quotas to work to benefit you as a user.

So if I understand correctly, a user can download, say, a 100 GB file in day 1 but then has to wait ~2 days (50 hours) to download anything else?