BetaArchive
https://www.betaarchive.com/forum/

Methods for preserving digital games?
https://www.betaarchive.com/forum/viewtopic.php?f=13&t=40818
Page 1 of 2

Author:  mrpijey [ Sun Feb 09, 2020 7:07 pm ]
Post subject:  Methods for preserving digital games?

Hello all!

I am once again looking for some ideas and expertise from the community.

As you all know digital downloads are becoming the norm more and more, and physical copies are going the way of the dinosaurs. This means that as a preservation community we need to look towards a way of properly preserving these too. And I wish members would start uploading stuff like Steam, EGS and Uplay libraries for future preservation as well, but the main issue here is finding a way to guarantee that members make proper copies and also be able to prove that they have these games without compromising privacy. With physical titles it's easy since all I need to do is ask for a photo of the physical item, and even the disc dump by itself is a good indication that the copy is original. But with digital copies it's much more difficult.

So, anyone have an idea? I would not want people to upload some second hand junk releases from some page, but these must be clean and unaltered copies from the digital stores, so we need to find a way for members to properly copy these and prove their authenticity as well.

Thank you.

Author:  yourepicfailure [ Sun Feb 09, 2020 8:21 pm ]
Post subject:  Re: Methods for preserving digital games?

One problem, for instance, is that Steam downloads are tied to user ID and won't run with any other user's client. This could pose a risk, especially with it possibly facilitating users attempting to find (and ask about) ways to circumvent.

However, with digital copies a big flag is whether or not the DRM has been tampered with. Digital copies have considerable DRM methods which can also determine whether the game itself has been tampered with, such as Denuvo.
We could ask users to upload photos of receipts from the provider, with the personal info blurred out.

Author:  zde [ Sun Feb 09, 2020 10:55 pm ]
Post subject:  Re: Methods for preserving digital games?

Well, as first tiny step we could make backup of GOG games. They offer same downloads for everyone, not digital marked, not DRMed... so comparing uploads from two members would be enough.

These are very small subset of all available games, though, but before Steam games or any other such games would be collected we would need to find a way to actually run them.

What's the point of "genuine" backup if it could only be used as long as Steam is around? If Steam is around and works you could just download everything from Steam!

The really sad story are mobile games: majority of them would not function without servers... and old versions usually don't work even if they are completely "genuine" thus it's not clear what could be done in principle...

Author:  Enaske [ Sun Feb 09, 2020 11:19 pm ]
Post subject:  Re: Methods for preserving digital games?

Well, don't steam has CRC Checksums? So you could just check for CRC Checksums if it is a propper release. I mean most of the people wont be able to use the Backups anyway, since the DRMs that are in place.

Author:  techosarusrex [ Sun Feb 09, 2020 11:29 pm ]
Post subject:  Re: Methods for preserving digital games?

well for some Windows games and apps that are free (as paid ones have DRM) you can download the package manually from a certain site

Author:  Darkstar [ Sun Feb 09, 2020 11:41 pm ]
Post subject:  Re: Methods for preserving digital games?

For Steam games, there is Goldberg Emulator (I won't link it here to not break any rules but it should be easy enough to find). Seems to work fine with most games.

As for proof, how about a screenshot of the Steam library showing the game in question? People could still try and fake it, or buy a game, make the screenshot and then get a refund, but the current proof for disc-based games can likewise be faked if you really want to...

Another option would be for people uploading stuff to add you to their contact list, then you can confirm the games without releasing any personal information.

Also, for flash-based online games, there's BlueMaxima's Flashpoint which could be mirrored on BA if you really wanted to (but it's already on many many archives so it's well-preserved)

Author:  PlyrStar93 [ Sun Feb 09, 2020 11:53 pm ]
Post subject:  Re: Methods for preserving digital games?

From what I know about Steam, it's kind of easy to show someone owns a game if the user sets profile and game library to public. Just pull up Steam web API to get a list of owned games, and find the appid in question.

Alternatively ask the user to show a screenshot of Steam client with that game, or console output of
Code:
licenses_for_app <appid>
or part of the licenses page of the account. Maybe a combination of them if one is not sufficient.

To make sure the upload is unaltered, we will have some problems if we don't rely on trust for now, unfortunately. Nothing really stops a user from adding some random junk into the folder before zipping and uploading, or modify an existing file in any way. If someone else happens to own the game, in some cases that user can tell right away if the upload is not original (e.g. difference in file size, total number of files, etc.) but it's not 100% foolproof and it requires some extra work on the checker.

Darkstar wrote:
People could still try and fake it, or buy a game, make the screenshot and then get a refund

Maybe set up a rule about uploading Steam games-- You must have owned the game for a period of at least beyond the refund timeframe, or something like that...

There will be several ways to show when the game is purchased, as well as how.

Author:  mibrab [ Mon Feb 10, 2020 12:15 am ]
Post subject:  Re: Methods for preserving digital games?

One thing to take in consideration with Steam games is the signature of steam_api.dll. If it has been signed by Valve, the DRM has not been circumvented and therefore it's very unlikely for the game to have been tampered with. (This would still be a huge assumption though.)

Also I don't think that showing that you own the game actually proves that the copy has not been tampered with, as there is nothing stopping you from replacing game files in a fully functional Steam copy. (apart from possibly the game DRM itself)

Author:  mrpijey [ Mon Feb 10, 2020 12:16 am ]
Post subject:  Re: Methods for preserving digital games?

yourepicfailure wrote:
One problem, for instance, is that Steam downloads are tied to user ID and won't run with any other user's client. This could pose a risk, especially with it possibly facilitating users attempting to find (and ask about) ways to circumvent.

However, with digital copies a big flag is whether or not the DRM has been tampered with. Digital copies have considerable DRM methods which can also determine whether the game itself has been tampered with, such as Denuvo.
We could ask users to upload photos of receipts from the provider, with the personal info blurred out.

This can be easily circumvented with Steam emulators, so that's not an issue. And it applies to "cracks" so users won't be given any emulators etc, but it can be done. Also remember, this is not about allowing people to run pirated stuff, but to preserve them. There will always be ways to bypass these protections, if not today then in the future.

Enaske wrote:
Well, don't steam has CRC Checksums? So you could just check for CRC Checksums if it is a propper release. I mean most of the people wont be able to use the Backups anyway, since the DRMs that are in place.

No checksums any user can browse and verify no. Internally yes since it has a means to check if your copy is correct, but not to the public as far as I know. And steam libraries work with various depots too so it would be difficult to gather all that info properly.

PlyrStar93 wrote:
From what I know about Steam, it's kind of easy to show someone owns a game if the user sets profile and game library to public. Just pull up Steam web API to get a list of owned games, and find the appid in question.

Well I won't ask members to share their libraries or set the libraries to public, that would be a no. AppID however would be required to know, but it would be hard to know if it was grabbed from their library or downloaded elsewhere. Regardless how we do it it would be cumbersome for the user too.

mibrab wrote:
One thing to take in consideration with Steam games is the signature of steam_api.dll. If it has been signed by Valve, the DRM has not been circumvented and therefore it's very unlikely for the game to have been tampered with. (This would still be a huge assumption though.)

Also I don't think that showing that you own the game actually proves that the copy has not been tampered with, as there is nothing stopping you from replacing game files in a fully functional Steam copy. (apart from possibly the game DRM itself)

The DLL by itself is no proof it's not been altered or some files are missing. And some Steam emulators can work without the DLL too so the original DLL can remain and the game can still be heavily modified.

Author:  PlyrStar93 [ Mon Feb 10, 2020 12:44 am ]
Post subject:  Re: Methods for preserving digital games?

mibrab wrote:
One thing to take in consideration with Steam games is the signature of steam_api.dll. If it has been signed by Valve, the DRM has not been circumvented and therefore it's very unlikely for the game to have been tampered with. (This would still be a huge assumption though.)

steam_api.dll and its digital signature have nothing to do with the other game files, as mrpijey pointed out. In addition there are games downloaded from Steam and do not come with steam_api.dll.

Author:  jackhenry [ Mon Feb 10, 2020 5:04 am ]
Post subject:  Re: Methods for preserving digital games?

As mentioned previously by other users, the issue of digital-only games being tied to their publisher's platform - be it Steam, Origin, Games for Windows Live or some other service, past or present - is something that should be addressed carefully as you do not want to spread personally identifying information, nor do you want to rely on service emulators or piracy cracks for such releases. I am unfortunately afraid that I have no idea how to deal with such methods of digital rights management in a legal fashion and thus I am not of much help with that particular issue. However, I have a simple and easy way to verify that what is being uploaded was purchased and is owned by the uploader so that you can be sure that the submission is not pure garbage.

All of the major publishers have support for various merchant processors as part of their storefront system, and in order to make any kind of purchase you must have a valid email address (which is also required for the opening of your game service account, naturally) and some form of payment be it a bank account, a credit and/or debit card, or a service such as PayPal. Tell me, when have you ever seen a financial institution of any capacity that does not keep some form of statements on file? When have you ever seen a digital games service not email you a receipt for your purchase, or keep a list of all your purchases somewhere in your account? I believe that in order to solve the issue of verifying if the uploader legally owns the game they are uploading, they should provide

- Their emailed receipt from the game service,
- A copy of their game service account purchase history,
- and, a copy of their financial institution's statements from the time said game purchase was made.

Naturally I believe that the uploader should take care to remove any sensitive and/or personally identifying information from their financial institution's statements; I'm sure that BetaArchive doesn't really care where the user physically lives, or what their bank account number is, or what other purchases the uploader made in that month, only that their statements show that they purchased and legally own the game. And of course, said statements should only be used for verification that the user legally owns the game that they are uploading to BetaArchive and should not be included as part of the game's re-release on the BetaArchive FTP Server.

However this brings along another set of questions to ask. Since BetaArchive strives for 100% accuracy when releasing software, how should BetaArchive release its abandonware games from digital game services? Certainly we can't just give members keys to released games on their digital game service of choice. I have seen that BetaArchive requires MDF or KryoFlux dumps of physical disks in the name of 100% accuracy as ISO and IMG dumps are imperfect and cannot dump discs anywhere near as well as said MDF or KryoFlux systems can. So with 100% accuracy in mind, how would BetaArchive intend to release digital games? Does Steam download files one at a time, or does it download a large compressed archive and decompresses the game when the download has been complete and verified? As I have alluded to in my thoughts on Microsoft Longhorn 4001, I believe that in order to be a true preservationist one needs to preserve software exactly as it was intended to be experienced.

Author:  yourepicfailure [ Mon Feb 10, 2020 5:13 am ]
Post subject:  Re: Methods for preserving digital games?

mrpijey wrote:
This can be easily circumvented with Steam emulators, so that's not an issue. And it applies to "cracks" so users won't be given any emulators etc, but it can be done.

Touche

mrpijey wrote:
Also remember, this is not about allowing people to run pirated stuff, but to preserve them. There will always be ways to bypass these protections, if not today then in the future.

I acknowledge that. However, I was attempting to highlight a possible increase in users who may not have read and/or understand the forum rules and motto fully and may wind up asking how to circumvent DRM.

Author:  Stannieman [ Sun Feb 16, 2020 6:51 pm ]
Post subject:  Re: Methods for preserving digital games?

Verifying the integrity from a single source indeed seems hard.
The first thing that came to mind to tackle this is some kind of AccurateRip-like system.
If multiple people own the game and all provide checksums of their own copy, then if the uploaded copy matches 90% of the others we can be quite certain it's not tampered with.

Author:  mrpijey [ Mon Feb 17, 2020 12:56 am ]
Post subject:  Re: Methods for preserving digital games?

Well, we could setup a system where an uploaded release is only verified as original if more members provide their copies so we can do a proper hash check. There will be minor differences in some Steam-specific files, but these can be easily identified and excluded. But we need to find a way to make it easy for users to dump their Steam/Uplay/Origin/MS Store/etc copies. All of them are pretty basic and only needs to be packed up, MS Store are more complicated as most copies are encrypted, but even this can be sorted out in one or another way.

Author:  Leon [ Mon Feb 17, 2020 7:42 am ]
Post subject:  Re: Methods for preserving digital games?

AFAIK in both Steam & EGS cases they use manifest files to hash file parts & verify the integrity of the game files, and I'm pretty sure there's tools available that can verify these manifests without needing Steam/EGS installed or logged in too.
(EPOC works for checking EGS games, but not really sure about Steam - there's like a million Steam related tools out there though, so I'm sure someone has made a manifest checker...)

Maybe uploading those manifests (and having them validate fine) could be a requirement for peoples uploads to be "original"-worthy?

I'm not sure whether the manifests themselves might contain user-specific data or not though - might be a good idea to strip them from the published release files, if manifest-checking becomes implemented.

E: Should note that in Steams case the manifest files are located in Program Files\Steam\depotcache\ folder, while EGS has them inside <gamedir>\.egstore\.

I did find a tool that seems like it can verify steam manifests (aptly named "manifestViewer") but not sure about linking it since it's hosted on a prominent Steam piracy site :/

Author:  mrpijey [ Mon Feb 17, 2020 10:58 am ]
Post subject:  Re: Methods for preserving digital games?

Well any files that would have personal data would need to be uploaded but it would not be published.

You can PM me this tool or a link and I will check it out, that would be great!

Thanks.

Author:  Leon [ Mon Feb 17, 2020 12:20 pm ]
Post subject:  Re: Methods for preserving digital games?

Oh I just found out that no-intro.org has a guide about checking/preserving Steam games: https://wiki.no-intro.org/index.php?tit ... eservation

The TINcft tool used there seems a lot easier to work with than the one I mentioned too, but I would mention that the "Obtaining Manifests" / "Download Depot" sections can probably be skipped if the game was downloaded through steam: you can just grab the manifests from the depotcache folder, and use the game files from steamapps/common/.

I don't see anything about EGS there though, so I'll PM you about that in a sec.

E: Also you might want to refer to SteamDB for any uploaded apps too, since the upload could be missing depots.
eg. the Morrowind depot list at https://steamdb.info/app/22320/depots/ has extra depots for German/French content, but my Morrowind copy download fresh from Steam seems to be missing those (I only have 22321_1293785437610275498.manifest in my depotcache folder)
I guess the only way for me to get them would probably be changing Steam language, or using the download_depot command mentioned on that wiki.

Hopefully any beta games probably wouldn't separate things into many different depots like this, but you should probably look out for them anyway.

Author:  jagotu [ Mon Feb 17, 2020 3:04 pm ]
Post subject:  Re: Methods for preserving digital games?

I somewhat looked into this and the way I understand this, the only thing that you really need to preserve a game is so-called "Depot Decryption Key" for that particular depot. Once you have that, you can easily download the manifests for said depot and decrypt all content.

While the Depot decryption keys are only requestable by accounts that have bought the game, they seem to be universal across user accounts.

So one way of contributing steam libraries would be providing you with depot decryption keys for all owned games and letting you download/decrypt/validate the whole thing.

One can probably use something like https://github.com/DoctorMcKay/node-steam-user/ with its getDepotDecryptionKey method to get the keys, and then a modified https://github.com/SteamRE/DepotDownloader (which would not request a depot key but use the saved one - modifications required around here: https://github.com/SteamRE/DepotDownloa ... er.cs#L542) to download and verify the content.

EDIT: I actually confirmed that I can anonymously download a game with just the knowledge of the depot decryption key.

Author:  Leon [ Mon Feb 17, 2020 7:38 pm ]
Post subject:  Re: Methods for preserving digital games?

jagotu wrote:
I somewhat looked into this and the way I understand this, the only thing that you really need to preserve a game is so-called "Depot Decryption Key" for that particular depot. Once you have that, you can easily download the manifests for said depot and decrypt all content.

While the Depot decryption keys are only requestable by accounts that have bought the game, they seem to be universal across user accounts.

So one way of contributing steam libraries would be providing you with depot decryption keys for all owned games and letting you download/decrypt/validate the whole thing.

One can probably use something like https://github.com/DoctorMcKay/node-steam-user/ with its getDepotDecryptionKey method to get the keys, and then a modified https://github.com/SteamRE/DepotDownloader (which would not request a depot key but use the saved one - modifications required around here: https://github.com/SteamRE/DepotDownloa ... er.cs#L542) to download and verify the content.

EDIT: I actually confirmed that I can anonymously download a game with just the knowledge of the depot decryption key.

Oh interesting, would be much better if something like this could work.

I had a quick look at node-steam-user's code, and it seems an auth-token is used when requesting manifest/depot chunks though? (search cdn.js for getCDNAuthToken to see where it's used)
You're sure you were able to download anonymously with just the depot key? Is it possible you tested with a game that was anonymous-download-enabled or something like that? Or maybe you had an auth-token cached already?

(It's too bad none of these tools have anything for loading supplied depot-keys built into them already, would really like to give this a try myself...)

Author:  jagotu [ Mon Feb 17, 2020 9:44 pm ]
Post subject:  Re: Methods for preserving digital games?

Leon wrote:
jagotu wrote:
I somewhat looked into this and the way I understand this, the only thing that you really need to preserve a game is so-called "Depot Decryption Key" for that particular depot. Once you have that, you can easily download the manifests for said depot and decrypt all content.

While the Depot decryption keys are only requestable by accounts that have bought the game, they seem to be universal across user accounts.

So one way of contributing steam libraries would be providing you with depot decryption keys for all owned games and letting you download/decrypt/validate the whole thing.

One can probably use something like https://github.com/DoctorMcKay/node-steam-user/ with its getDepotDecryptionKey method to get the keys, and then a modified https://github.com/SteamRE/DepotDownloader (which would not request a depot key but use the saved one - modifications required around here: https://github.com/SteamRE/DepotDownloa ... er.cs#L542) to download and verify the content.

EDIT: I actually confirmed that I can anonymously download a game with just the knowledge of the depot decryption key.

Oh interesting, would be much better if something like this could work.

I had a quick look at node-steam-user's code, and it seems an auth-token is used when requesting manifest/depot chunks though? (search cdn.js for getCDNAuthToken to see where it's used)
You're sure you were able to download anonymously with just the depot key? Is it possible you tested with a game that was anonymous-download-enabled or something like that? Or maybe you had an auth-token cached already?

(It's too bad none of these tools have anything for loading supplied depot-keys built into them already, would really like to give this a try myself...)


Yeah, I noticed that. SteamKit2 also has that. I think it has to do with downloading from private branches though, as the actual Steam Client downloads public chunks completely unauthenticated

I gutted the CDN token stuff from DepotDownloader and fed it a hardcoded depotkey with some fugly reflection stuff and was able to download the binaries just fine. I confirmed that there are no authentication tokens stored or sent. Tested with both Tomb Raider I and Spyro Reignited Trilogy and it worked fine.

Author:  mrpijey [ Tue Feb 18, 2020 8:21 am ]
Post subject:  Re: Methods for preserving digital games?

So how would the process work then? Can someone download the required tools and test this out, and describe what a user would need to do to download a particular title?

Author:  jagotu [ Tue Feb 18, 2020 10:55 am ]
Post subject:  Re: Methods for preserving digital games?

The process would work like this:

An user runs a special tool (that we would need to create but is definitely technically possible), logins to his Steam account and the tool walks through his complete Steam library, requests depot decryption keys for each of his games and dumps them to a text file.

The user uploads only the depot decryption keys file to BA FTP.

You take these decryption keys and download the game directly from Steam's CDN (using for example modified DepotDownloader) with blazing fast speed and 100% authenticity (as you got the data yourself).

I can create said tools and modifications in the beginning of march, unless somebody finds the time earlier.

Author:  mrpijey [ Tue Feb 18, 2020 8:38 pm ]
Post subject:  Re: Methods for preserving digital games?

Alright, that sounds like a plan then! Great! Will save a lot of time too since I can get everything at high speed, and properly.

Do you know if Steam stores the version of the game too? Since we will have to maintain a list of archived games with the version number, and any updates that might come along with it.

Do these depot decryption keys expire after a time? Or can they be kept (privately of course) and used to grab future updates?

Author:  Darkstar [ Tue Feb 18, 2020 10:32 pm ]
Post subject:  Re: Methods for preserving digital games?

jagotu wrote:
The process would work like this:

An user runs a special tool (that we would need to create but is definitely technically possible), logins to his Steam account and the tool walks through his complete Steam library, requests depot decryption keys for each of his games and dumps them to a text file.

The user uploads only the depot decryption keys file to BA FTP....


How does having the depot key prove anything? If they are not unique, there probably are already databases of those keys online, where anyone could copy/paste keys and "prove" that he owns a particular game. How does that prove anything that a simple screenshot of the game library doesn't?

In my opinion, the only really trustworthy source that can be queried whether a particular user has or does not have a certain game, is Steam itself, i.e. the (public) profile or some other API that lets others query the game list of a certain user ID...

Author:  jagotu [ Tue Feb 18, 2020 10:45 pm ]
Post subject:  Re: Methods for preserving digital games?

Darkstar wrote:
How does having the depot key prove anything? If they are not unique, there probably are already databases of those keys online, where anyone could copy/paste keys and "prove" that he owns a particular game. How does that prove anything that a simple screenshot of the game library doesn't?

In my opinion, the only really trustworthy source that can be queried whether a particular user has or does not have a certain game, is Steam itself, i.e. the (public) profile or some other API that lets others query the game list of a certain user ID...


I'm sorry, but what exactly is the problem you are trying to solve? I was solving a problem "make sure that files ending on the FTP are authentic", and I believe my proposition serves that purpose. Proving game ownership is a completely different issue.

mrpijey wrote:
Alright, that sounds like a plan then! Great! Will save a lot of time too since I can get everything at high speed, and properly.

Do you know if Steam stores the version of the game too? Since we will have to maintain a list of archived games with the version number, and any updates that might come along with it.

Do these depot decryption keys expire after a time? Or can they be kept (privately of course) and used to grab future updates?


From the way I understand now, proper experimental verification pending:

Basically there's one manifest version for each game version, !BUT! Steam only gives you the latest manifest ID. So if you want to go back in history, you need to have previous knowledge of the manifest ID of the previous version. That's where sites like steamdb.info come in handy. The manifest unfortunately doesn't give any further indication of a human-readable version number (the ID can be something like 2823514620272099070), so old ways like checking the main exe versions etc. need to be used.

As for whether the depot keys change, my guess is they shouldn't, as all the downloaders use just the appID and depotID when requesting the keys, so it shouldn't be bound to a specific manifestID and as such to a specific version. But time will tell.

Page 1 of 2 All times are UTC [ DST ]
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/