BetaArchive Logo
Navigation Home Database Screenshots Gallery Image Uploader Server Info FTP Servers Wiki Forum RSS Feed Rules Please Donate
UP: 1d, 9h, 12m | CPU: 10% | MEM: 4795MB of 8191MB used
{The community for beta collectors}

Post new topic Reply to topic  [ 45 posts ]  Go to page Previous  1, 2
Author Message
 PostPost subject: Re: Methods for preserving digital games?        Posted: Tue Feb 18, 2020 11:05 pm 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
8084
This is where my concern comes from too, if the depot keys are not unique then why don't we already have vast databases listing every key for every title already? If they are unique they are tied to the user account, which probably makes it difficult for an another user to download the games.

But it would be nice with some scripts or tools to work with and see how it works, so we can design an easy way for members to extract the required data so it can be submitted to me for download, or a means for the user to download the complete title themselves for upload.

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter


Top  Profile  WWW
 PostPost subject: Re: Methods for preserving digital games?        Posted: Tue Feb 18, 2020 11:21 pm 
Reply with quote
FTP Access
User avatar
Offline

Joined
Mon Feb 04, 2013 5:03 pm

Posts
514

Location
Czechia

Favourite OS
Development Release #5
mrpijey wrote:
This is where my concern comes from too, if the depot keys are not unique then why don't we already have vast databases listing every key for every title already? If they are unique they are tied to the user account, which probably makes it difficult for an another user to download the games.

But it would be nice with some scripts or tools to work with and see how it works, so we can design an easy way for members to extract the required data so it can be submitted to me for download, or a means for the user to download the complete title themselves for upload.


I'll skip that first question as asking "why isn't there something yet?" can only really be answered with "because nobody thought of it and/or took the time to do it".

Anyway, I was able to take this random depot key for Arma 2 I found somebody posted on steam forums back in 2013 and use it to download the latest game exe. I don't even own Arma 2.

Image
Image

Btw. I guess "vast databases listing every key for every title already" would be considered very illegal.

_________________
Windows TEN - Totally Erroneous Numbering
Always watching you...


Top  Profile  WWW
 PostPost subject: Re: Methods for preserving digital games?        Posted: Tue Feb 18, 2020 11:41 pm 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
8084
And why would it be illegal? If the key isn't personal and tied to an account it's just a bunch of random numbers, and no more illegal than providing depot IDs they already have databased.

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter


Top  Profile  WWW
 PostPost subject: Re: Methods for preserving digital games?        Posted: Wed Feb 19, 2020 11:14 am 
Reply with quote
Offline

Joined
Wed Nov 13, 2019 7:31 pm

Posts
32
mrpijey wrote:
This is where my concern comes from too, if the depot keys are not unique then why don't we already have vast databases listing every key for every title already?
Why don't we have wast databases for Microsoft downloads which are not even encrypted?

My guess would be: it's a lot of hassle to support such databases. Keys may change when game is updated - and games are updated all the time on Steam. Not a problem if you only need to track one or two games (they wouldn't update that often), but serious problem for "wast database".

Should probably work fine for the approach you are suggesting: when key is sent to you for you to download game and then is not used anymore.

mrpijey wrote:
And why would it be illegal? If the key isn't personal and tied to an account it's just a bunch of random numbers, and no more illegal than providing depot IDs they already have databased.
There's subtle difference. Depot ID is shown to anyone on the Steam website. You don't even need to own the game to see it.

Encryption key, on the other hand, is only shown to owner of the title - thus it's definitely may be declared illegal as per DMCA. DMCA doesn't say anything about quality of "technical protection measures" which you are not supposed to circumvent. Ultimately court would have to decide if that's "just a bunch of random numbers" or "technical protection measure". There were already decisions both way (when something really easy to circumvent was still accepted as "technical protection measure" and when something was declared is too simple to circumvent to qualify) thus we couldn't say one way or another without court.

Think about Blu-Ray movies: all the movies are encrypted with the same key and if you would know that key - you can easily decrypt that movie. But! That key is not stored anywhere on an SSD of a system which plays Blu-Ray movie and actually there are a lot of protection to make it's extraction hard. Thus key for Blu-Ray movie is definitely a protection scheme.

But what about all these files which are uploaded to Mega with decryption keys on pubic forums? These are definitely not "technical protection measure" - they are just used to stop Mega from snooping inside of the archive.

Steam is somewhere in the middle: key is only supposed to be given to people who own the game, yet it's kept in easy to access text file... who knows which way court will decide if it would come to that?


Top  Profile
 PostPost subject: Re: Methods for preserving digital games?        Posted: Wed Feb 19, 2020 11:39 am 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
8084
There's a major difference on so many levels between a database of binaries and a database of hexcode numbers. So your comparison is massively flawed. And databases are designed to be updated, see the steamdb.info page, Wikipedia, heck, even our own database. It all contains text, and no binaries. And the key has been proven to not be account bound either.

Your second point is more valid, however DMCA laws only apply to ONE country. So there would technically (and most likely legally) no issues hosting it elsewhere. Remember, US laws are not global.
Bluray movies are also easy to decrypt as they use a few, by now well known and widely distributed, decryption keys. But I can simply accept that the Steam encryption keys are not indexed in a public database to prevent any potential legal issues as it's a grey area - at one hand it's just a sequence of numbers, not tied to a personal account and therefore not unique other than to the title itself, but on the other hand it can be used to download, NOT RUN, a title you're not entitled to run. But then the same could be applied to BetaArchive and practically any abandonware and file distribution site out there as they are distributing binaries without a proper license, and even Google itself for providing means to read information you're officially not entitled to. Remember also that Steam themselves provide open source code to handle Steam downloads so if this had been an issue I think Steam would not have been so forthcoming with providing solutions for easy downloads, authentic or otherwise.

But regardless, if we would move on with trying to preserve digital games none of these keys would be shared in any way through BetaArchive. It was just an interesting question considering that the keys are not unique to an account.

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter


Top  Profile  WWW
 PostPost subject: Re: Methods for preserving digital games?        Posted: Wed Feb 19, 2020 4:21 pm 
Reply with quote
Donator
User avatar
Offline

Joined
Fri May 14, 2010 1:29 pm

Posts
891

Location
Southern Germany

Favourite OS
IRIX 5.3
jagotu wrote:
I'm sorry, but what exactly is the problem you are trying to solve? I was solving a problem "make sure that files ending on the FTP are authentic", and I believe my proposition serves that purpose. Proving game ownership is a completely different issue.

The problem I'm trying to solve is that up to now, BA has always required proof that the user who uploads something has the physical item in his posession. That's where the scan requirement comes from. For digital-only games there is, of course no physical proof. My point is that saying "I want to upload game X, and here is the the decryption key to prove that I own it" will not work. That's all.

The other concern I have is that giving away the decryption key is an inversion of control. Until now, users decide what software they want to share with/upload to BA individually, and that freedom to not share something if you don't want to has been held in high regard. By providing a decryption key they allow BA to download and decrypt all future versions of that game too, even if, for example, the user in question has already left BA for some reason or another. There is no way for the user to "take back" the key.

But I agree that both of these points are rather minor in the Grand Scheme Of Things. Just something to be aware of

_________________
I upload stuff to archive.org from time to time. See here for everything that doesn't fit BA


Top  Profile
 PostPost subject: Re: Methods for preserving digital games?        Posted: Wed Feb 19, 2020 8:23 pm 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
8084
The major difference here will be that a Steam title will only be allowed to be submitted by a single user, and no other users will be allowed to submit the same game as the "copies" are all identical and uses the same encryption keys.

Since the encryption key is not personal and tied to the user's account I don't see this as any issues, since I wouldn't download the user's game or license. And members don't have any control of the releases today either as they can't revoke a release from the FTP. And the key will only be used by a single person. It would also be a lot easier for a member to submit digital games (Steam games that is) since they don't have to do any of the packaging or uploading. All they need is to provide the hexadecimal key and that's it.

But some proper guidelines around this will have to be established also so it's not abused, as we want to work together to preserve software and not abuse it to get access to the FTP etc.

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter


Top  Profile  WWW
 PostPost subject: Re: Methods for preserving digital games?        Posted: Wed Feb 19, 2020 9:08 pm 
Reply with quote
Donator
User avatar
Offline

Joined
Fri May 14, 2010 1:29 pm

Posts
891

Location
Southern Germany

Favourite OS
IRIX 5.3
With all these bundles and steam keys that sell for a dollar or even less, I fear BA will soon be swamped with low-grade indie games by people who would do anything to gain FTP access.

I vote for not counting Steam games towards FTP access because of that, to not fill BA with crappy games.

_________________
I upload stuff to archive.org from time to time. See here for everything that doesn't fit BA


Top  Profile
 PostPost subject: Re: Methods for preserving digital games?        Posted: Wed Feb 19, 2020 10:54 pm 
Reply with quote
Offline

Joined
Wed Nov 13, 2019 7:31 pm

Posts
32
Before these would be good for FTP access the process need to be tested, verified, bugs ironed out, etc. I think eventually they would give FTP access... but not any time soon.


Top  Profile
 PostPost subject: Re: Methods for preserving digital games?        Posted: Wed Feb 19, 2020 11:00 pm 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
8084
Darkstar wrote:
With all these bundles and steam keys that sell for a dollar or even less, I fear BA will soon be swamped with low-grade indie games by people who would do anything to gain FTP access.

I vote for not counting Steam games towards FTP access because of that, to not fill BA with crappy games.

Naturally we wouldn't save every piece of trash existing on Steam. Primarily I will focus on AAA-games and well known AA-games. So don't worry, we won't archive every piece of Flappy Bird junkclone that exists on Steam or any other platform :).

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter


Top  Profile  WWW
 PostPost subject: Re: Methods for preserving digital games?        Posted: Thu Feb 20, 2020 1:01 pm 
Reply with quote
Offline

Joined
Wed Nov 13, 2019 7:31 pm

Posts
32
Well, flappy bird itself must definitely be preserved. It's stupid game but it has historical significance, for sure.

But I'm sure there are a lot of people who stashed it already.


Top  Profile
 PostPost subject: Re: Methods for preserving digital games?        Posted: Sat Feb 29, 2020 12:50 am 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
8084
I've been starting to archive Steam games by downloading them through the depot downloader, but I've come to a realisation that it's completely impractical to keep every build of every game possible, at least when it comes to the RTM games. To keep the sets within tolerable sizes I will only keep the latest build of each game, with some exceptions of course. One exception for example would be Grand Theft Auto 4 as they are removing multiplayer functionality with the newest patches. But otherwise the latest version would be the best one to preserve.

How did I come to this decision? Well, here are two examples:

- ARK: Survival Evolved. Each set takes up approximately 300GB, so 1.5TB just for that game.
- Call of Duty: Modern Warfare. The game takes up 250GB per set, and I have 8 sets of it now.... so that's 2TB just for this game.

See the pattern? Apply this to thousands of games and we're talking not TB, but PBs of data just for keeping every version. And even deduplication would be massively impractical for this unless we would have some kind of intelligent SVF system that could handle whole sets in a smart way, but that doesn't exist.

It's simply not practical to keep every possible version for these games, and since Steam, EGS etc don't release individual patches but whole game sets it's impossible to grab the patches alone and keep them, so I am forced to download whole sets every time there's an updated version.

This will only apply to digitally distributed RTM versions. I will make some exceptions but these should not be many as usually a publisher patches up a game until they decide to not patch it anymore, which by then should be the best available version of the game. And unless they remove critical features (such as in the GTA IV case) there wouldn't be any need to keep every version of a game. "Generational" games such as World of Warcraft, EVE Online, Elder Scrolls Online etc which gets large content patches and changes "generation" would also be such exception where I would perhaps keep the first and last version of each gen, but it would be difficult to keep track of this, and I usually then save the first version or any version I can get my hands on at the time. In the case of World of Warcraft etc we also have the physical media to archive which retains the "retail" copy of the game.

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter


Top  Profile  WWW
 PostPost subject: Re: Methods for preserving digital games?        Posted: Mon Mar 09, 2020 6:18 pm 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
8084
Been working some the Steam script to properly preserve Steam games, this is a sample of the file structure it outputs:

Code:

+---Age of Empires Online (4.00.6148) (2020-02-28) [#105430]
|   \---media_depot
|           105431 (Age Of Empires Online Depot).rar
|           
+---Age of Wonders 3 (2020-02-29) [#226840]
|   \---media_depot
|           226841 (Age of Wonders III).rar
|           226843 (Age of Wonders III - Windows).rar
|           226844 (Age of Wonders III - OSX).rar
|           226845 (Age of Wonders III - Linux).rar
|           228983 (VC 2010 Redist).rar
|           228990 (DirectX Jun 2010 Redist).rar
|           
+---Alan Wake (2020-02-29) [#108710]
|   \---media_depot
|           108712 (Alan Wake ThirdParty).rar
|           108714 (Alan Wake german).rar
|           108715 (Alan Wake tchinese).rar
|           108716 (Alan Wake spanish).rar
|           108717 (Alan Wake french).rar
|           108718 (Alan Wake italian).rar
|           108719 (Alan Wake Japanese).rar
|           108720 (Alan Wake koreana).rar
|           108721 (Alan Wake Polish).rar
|           108722 (Alan Wake Russian).rar
|           108723 (Alan Wake ExesAndShaders).rar
|           108724 (Alan Wake Videos).rar
|           108725 (Alan Wake Datas).rar
|           108728 (Alan Wake Czech).rar
|           
+---Amnesia - A Machine for Pigs (2020-02-29) [#239200]
|   \---media_depot
|           228983 (VC 2010 Redist).rar
|           239201 (Machine for Pigs Content).rar
|           239202 (Machine for Pigs Windows).rar
|           239203 (Machine for Pigs OS X).rar
|           239204 (Machine for Pigs Linux).rar
|           
+---Amnesia - The Dark Descent (2020-02-29) [#57300]
    \---media_depot
            228983 (VC 2010 Redist).rar
            57301 (AmnesiaTDDData).rar
            57302 (AmnesiaTDDWindows).rar
            57303 (AmnesiaTDDMacOS).rar
            57304 (AmnesiaTDDLinux).rar


The script doesn't require the Steam client to be active or even installed as it uses its own application to access and download from the Steam network. It saves the depot name as Steam presents it, and the main title has the #AppID included too. All the depot keys are stored with the release as well so updates can be downloaded without the need for access to the account. The keys are unique for the depot but not for the user account.
I still need help with storing any depots that are not accessible so these can be downloaded or added by an another user, so the script is not yet ready for public release.

Steam must be applying some heavy duty deduplication as the depots are in most cases extremely inefficiently used (videos etc are duplicated between Mac, Win and Linux releases even when they don't have to be, and they should have been put into a separate depot by Steam).

But I am running the script now to archive my own Steam library for testing etc and it works fine. There are some further tweaks and code optimizations I can make, but at the moment I want to get everything to work first.

I am going to build a new set of tool pack to include everything needed to preserve games, regardless if it's digital or physical. If any users knows of any smart ways of automating Origin, Uplay etc archiving let me know.

To be continued...

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter


Top  Profile  WWW
 PostPost subject: Re: Methods for preserving digital games?        Posted: Wed Mar 11, 2020 1:22 am 
Reply with quote
Donator
User avatar
Offline

Joined
Fri May 14, 2010 1:29 pm

Posts
891

Location
Southern Germany

Favourite OS
IRIX 5.3
Okay, I'm starting to see the value in this. That's really awesome so far.

Could we also preserve the artwork that comes with each game, in the form of trading cards and badges etc.?

Trading Cards are virtual cards that you can randomly collect to unlock new artwork (one high-res image for every trading card) and other things (background images, custom emojis, etc. for each full set of trading cards).
I think especially the artwork would be worth saving, since some games have pretty extensive collectable artwork (dozens of high-res pictures), and some of these are new or unique and made especially for the Steam release of the game.

There are already databases of these artworks (e.g. this one) which contain previews of the full-size images (just scaled down and not saveable through the browser) that could be crawled. It would be nice to have these media assets preserved with the games

_________________
I upload stuff to archive.org from time to time. See here for everything that doesn't fit BA


Top  Profile
 PostPost subject: Re: Methods for preserving digital games?        Posted: Wed Mar 11, 2020 9:34 am 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
8084
I don't believe so, these trading cards are not part of the game but part of your Steam account, so they are not relevant to the preservation of the game. There is no way to get these cards without have actually earned them, and I don't believe there's any way to even extract the artwork without some very advanced account hacking etc, something I am not willing to go into. But the original point stands, these are not part of the game so they are not relevant to the game.

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter


Top  Profile  WWW
 PostPost subject: Re: Methods for preserving digital games?        Posted: Wed Mar 11, 2020 10:59 am 
Reply with quote
Donator
User avatar
Offline

Joined
Fri May 14, 2010 1:29 pm

Posts
891

Location
Southern Germany

Favourite OS
IRIX 5.3
mrpijey wrote:
I don't believe so, these trading cards are not part of the game but part of your Steam account, so they are not relevant to the preservation of the game. There is no way to get these cards without have actually earned them, and I don't believe there's any way to even extract the artwork without some very advanced account hacking...


Well, the link I posted shows how to "extract" them. I'm humbled by you thinking I'm an "advanced hacker" who deoes "advanced account hacking", but really, I'm not... it's just jpg files on the CDN of Steam, you get the direct link from the steam database itself (similar to how you get the decryption key). To be able to "see" these images in the Steam client, yes, you need to earn them, but in the end it's just jpegs on a server.

And if the digital artwork is not part of a digital game, then by the same argument, "physical" artwork (CD scans, covers, etc.) should not be part of a "physical" game. On the contrary, I would suggest that *all* digital assets from the Steam store are part of the game. That means the page header image, the screenshots and videos that showcase the game, etc.

_________________
I upload stuff to archive.org from time to time. See here for everything that doesn't fit BA


Top  Profile
 PostPost subject: Re: Methods for preserving digital games?        Posted: Wed Mar 11, 2020 3:05 pm 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
8084
Physical artwork (covers etc) are part of the product, they are designed and made by the publisher of the game and included as part of the package since the physical disc comes in that very same package along with the manual and additional content, designed to be shipped with that package. Steam achievements, artwork, cards etc are not part of the product as this is part of the Steam service, solely beneficial for Steam and its users and not the publishers of the game who only published the game to Steam, paid a sum of money to Steam to have it published. So I don't agree with you as we're archiving the publishers game distributed by Steam, we're not archiving the Steam service itself (with all its artwork etc). There's a major difference. I would have agreed with you if the same artwork and trading cards would have been shipped with the Uplay or GoG version of the same game, which they are not.

We can of course argue the value of archiving these trading cards separately, but that would be a completely different project requiring completely different tools. If you're up to the challenge to make these tools then go ahead and we can look into how to preserve them too, but then that would be a different kind of preservation (Steam service preservation) and not part of the actual game, as these trading cards etc are not part of the main product (the game).

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter


Top  Profile  WWW
 PostPost subject: Re: Methods for preserving digital games?        Posted: Wed Mar 11, 2020 8:18 pm 
Reply with quote
Donator
User avatar
Offline

Joined
Fri May 14, 2010 1:29 pm

Posts
891

Location
Southern Germany

Favourite OS
IRIX 5.3
I agree that Steam is a cutthroat business and that they take far too large cuts for publishing games through it (but there's no actual alternative to it so publishers will pay any price to get their games on Steam). I just was not aware that the financial aspect of distribution is related to preservation. But of course I accept your decision and I won't bring up that matter (and similar ones involving similarly greedy online services) again, and sorry for interrupting the flow of this thread.

Anyway, during lunch break today I wrote a small script to grab all the artwork from the Steam CDN, so should you ever change your mind just let me know and I will share that script with BA

_________________
I upload stuff to archive.org from time to time. See here for everything that doesn't fit BA


Top  Profile
 PostPost subject: Re: Methods for preserving digital games?        Posted: Wed Mar 11, 2020 9:09 pm 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
8084
You can share it if you want and I can have a look at it and see if I can add it to my BA scripts as an optional feature to be used by the members once the script pack is released.

And the aspect of distribution is relevant when it comes to bundling stuff that is not part of the product. Since the Steam artwork is part of Steam and not the game itself it's not part of the game, and therefore will not be preserved with the game. We can however discuss if it's worth saving the Steam artwork and how it should be saved, but I think that would be out of the scope of what we're preserving on BA as we're focused on the products themselves, not the distribution platforms or infrastructure behind it.

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter


Top  Profile  WWW
 PostPost subject: Re: Methods for preserving digital games?        Posted: Thu Mar 12, 2020 12:53 am 
Reply with quote
Donator
User avatar
Offline

Joined
Fri May 14, 2010 1:29 pm

Posts
891

Location
Southern Germany

Favourite OS
IRIX 5.3
I totally understand where you're coming from. Your view makes sense. But still, for me, the Steam artwork belongs to the game, since it was created by the game designer/developer/artists, and I don't believe it matters on what platform/publishing system that artwork is used. I see it more like an artbook that comes with some physical games. But again, I can see the reasoning behind your point of view as well, it's just different from mine.

I merely wanted to bring a point up for discussion that I thought might have been overlooked. It was considered, and ultimately rejected, and I'm fine with that. At least I know that someone else gave it some thought :)

_________________
I upload stuff to archive.org from time to time. See here for everything that doesn't fit BA


Top  Profile
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 45 posts ]  Go to page Previous  1, 2




Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  

All views expressed in these forums are those of the author and do not necessarily represent the views of the BetaArchive site owner.

Powered by phpBB® Forum Software © phpBB Group

Copyright © 2006-2020

 

Sitemap | XML | RSS