Authenticating releases

New news and release discussion.
mrpijey
User avatar
Administrator
Posts: 8178
Joined: Tue Feb 12, 2008 5:28 pm
Contact:

Authenticating releases

Post by mrpijey »

In the light of recent discussions about authentic floppy releases etc I've decided (and I don't know why I didn't implement this in the first place) to tag all the dumped releases that has been verified to come from original media. This to make it simpler for members to not only know what releases are 100% true, but also to know what to replace when they get hold of a better release.

To furfill the verification tag three requirements has to be met:
  1. The product has to be dumped from original media*
  2. Scans has to be provided of the media, and if possible the box cover, CD/DVD cover, the scans has to be in full resolution (300dpi or higher), scanned with a scanner (no cellphone pics or Google image downloads etc) and unaltered, so no watermarks, tags or any markings. Scans are to be provided in PNG, TIFF or TGA so they can be cleaned up and optimized before release.
  3. The product has to be unaltered in any way, no extra files, serials etc. Such files will disqualify the offer and it will be processed as non-verified. Write protect any floppies before dumping.
*In some cases the "media" is provided from Internet (such as downloadables from Adobe, Microsoft etc), if you provide these you must provide the original link of where you got it. If the source can't be verified then the release will be processed as a regular non-verified release.

Valid tools for dumping the media is WinImage and rawrite (for floppies), ImgBurn, CDRWin, CloneCD and Disk Utility (for Mac, use "DVD/CD master" as image format) (for disc media). Do not use tools such as UltraISO, WinISO, Nero etc as they are known to alter the images. Only true raw 1:1 dumps are accepted, no custom images. Use equivalent tools for the Linux platform if you must (such as dd etc).

All this is an attempt to properly catalogue and archive the most authentic release of a product we can. We have lots of releases now and I will make an attempt to verify each of them, a lot of them has already been dumped and scanned by me so these will be marked, same goes with user uploads that came with floppy scans etc.

If you got any questions don't be afraid to ask.
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter

x010
Staff
Posts: 1084
Joined: Thu Jun 13, 2013 4:46 pm
Contact:

Re: Authenticating releases

Post by x010 »

Can we use 7-ZIP ,WinRAR for compressing the images?
What do you mean by 'write protect the floppies'?
Why should we write-protect them?

DeFacto

Re: Authenticating releases

Post by DeFacto »

1. This was already answered numerous times: yes, you can.
2. There is usually a switch on the floppy, which makes the floppy read-only (=can only be read, but not written to).
3. Read what mrpijey wrote; certain OSes and programs tend to modify the floppies/images unless they're write protected.

mrpijey
User avatar
Administrator
Posts: 8178
Joined: Tue Feb 12, 2008 5:28 pm
Contact:

Re: Authenticating releases

Post by mrpijey »

Yes, WinRAR or 7z is fine since I recompress them anyway before release.

And write protect the floppies to prevent alterations to the header or access time stamps of the floppy. There's some discussions about it here. There's simply no reason to keep them writeable anyway.
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter

3155ffGd
User avatar
FTP Access
Posts: 391
Joined: Wed May 02, 2012 12:57 am

Re: Authenticating releases

Post by 3155ffGd »

For files which are just missing boxart, can it still be uploaded to the FTP and added to the existing archive?

x010
Staff
Posts: 1084
Joined: Thu Jun 13, 2013 4:46 pm
Contact:

Re: Authenticating releases

Post by x010 »

DeFacto wrote:1. This was already answered numerous times: yes, you can.
2. There is usually a switch on the floppy, which makes the floppy read-only (=can only be read, but not written to).
3. Read what mrpijey wrote; certain OSes and programs tend to modify the floppies/images unless they're write protected.
1 - I meant the floppy contents.
mrpijey wrote:Yes, WinRAR or 7z is fine since I recompress them anyway before release.

And write protect the floppies to prevent alterations to the header or access time stamps of the floppy. There's some discussions about it here. There's simply no reason to keep them writeable anyway.
Does that mean that we should keep the floppy read-only?

DeFacto

Re: Authenticating releases

Post by DeFacto »

x010 wrote:1 - I meant the floppy contents.
You must make floppy images using WinImage or some other program. Then you must compress the images in a RAR/7Z/ZIP archive. You shouldn't just take the floppy contents and make a folder dump; these are generally rejected if an image can be made.
x010 wrote:Does that mean that we should keep the floppy read-only?
Unless you want to modify them, then yes.

GL1zdA
User avatar
FTP Access
Posts: 496
Joined: Thu Dec 03, 2009 3:32 pm
Location: Warsaw, PL

Re: Authenticating releases

Post by GL1zdA »

Will floppies with IHC volume labels or last access date set (not the last modified timestamp) be marked as "verified"? Will you try to "purify" them be setting the volume labels/timestamps by hand (I know you recreate floppies from extracted setups)?
Image

AlphaBeta
User avatar
Donator
Posts: 2137
Joined: Sun Aug 12, 2012 4:33 pm
Location: Czechia

Re: Authenticating releases

Post by AlphaBeta »

GL1zdA wrote:Will floppies with IHC volume labels or last access date set (not the last modified timestamp) be marked as "verified"? Will you try to "purify" them be setting the volume labels/timestamps by hand (I know you recreate floppies from extracted setups)?
No.
AlphaBeta, stop brainwashing me immediately!

Image

mrpijey
User avatar
Administrator
Posts: 8178
Joined: Tue Feb 12, 2008 5:28 pm
Contact:

Re: Authenticating releases

Post by mrpijey »

3155ffGd wrote:For files which are just missing boxart, can it still be uploaded to the FTP and added to the existing archive?
What do you mean? If you can upload just the boxart to an existing archive? Yes, you can do that, but it won't be verified as authentic since I would still not know the files really match that particular media.
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter

3155ffGd
User avatar
FTP Access
Posts: 391
Joined: Wed May 02, 2012 12:57 am

Re: Authenticating releases

Post by 3155ffGd »

I'm in this case referring to a file I uploaded previously. I don't know why I should send all those 2 gigabytes through my (quite limited) intertubes again when just the boxart would do it as well. I guess I can understand the point if it's someone else's contribution, but I meant boxart to my own files.

mrpijey
User avatar
Administrator
Posts: 8178
Joined: Tue Feb 12, 2008 5:28 pm
Contact:

Re: Authenticating releases

Post by mrpijey »

Well, in your case just send me the boxart/media scans and tell me which release they belong to and I'll match it up.
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter

kinokijuf
FTP Access
Posts: 216
Joined: Mon May 09, 2011 5:43 pm
Location: Poland

Re: Authenticating releases

Post by kinokijuf »

mrpijey wrote:Valid tools for dumping the media is WinImage and rawrite (for floppies), ImgBurn, CDRWin, CloneCD and Disk Utility (for Mac, use "DVD/CD master" as image format) (for disc media).
Add dd to the list.
Nt 3.50: GOOD! Nt 3.51: THIS SHOULD BE A SERVICE PACK. Nt 4: NO COMMENT. Windows 2000: GREAT! Windows XP: BAD! Windows Vista: GREAT! Windows 7: SO-SO. Windows 8: BAD!
See? There’s no pattern here. I’m not interested in your conspiracy theories.

DeFacto

Re: Authenticating releases

Post by DeFacto »

It's mentioned in the next sentence as a Linux alternative...

kinokijuf
FTP Access
Posts: 216
Joined: Mon May 09, 2011 5:43 pm
Location: Poland

Re: Authenticating releases

Post by kinokijuf »

Ah. Nevertheless, it is also available on other platforms.
Nt 3.50: GOOD! Nt 3.51: THIS SHOULD BE A SERVICE PACK. Nt 4: NO COMMENT. Windows 2000: GREAT! Windows XP: BAD! Windows Vista: GREAT! Windows 7: SO-SO. Windows 8: BAD!
See? There’s no pattern here. I’m not interested in your conspiracy theories.

RedStripes
FTP Access
Posts: 28
Joined: Sun Sep 23, 2012 1:09 pm

Re: Authenticating releases

Post by RedStripes »

For 800k/400k Mac floppies an old Mac with built-in floppy drive (or external from Apple) is needed because of hardware differences. In this case only Disk Copy 4.2 should be used, later versions could destroy important information.

Using "DVD/CD master" format on Disk Uility will produce a raw image of a 1.44MB floppy disk.

Darkstar
User avatar
Donator
Posts: 918
Joined: Fri May 14, 2010 1:29 pm
Location: Southern Germany

Re: Authenticating releases

Post by Darkstar »

How about IPF files for older/exotic systems like the Mac? You can create them on the PC with a KryoFlux board. Since they are as close as you can get to the physical media, these should also be accepted as authentic.
I upload stuff to archive.org from time to time. See here for everything that doesn't fit BA

mrpijey
User avatar
Administrator
Posts: 8178
Joined: Tue Feb 12, 2008 5:28 pm
Contact:

Re: Authenticating releases

Post by mrpijey »

It's also a matter of usability, only four emulators support IPF and not many other applications support it. There's no point storing things in a format that can't be easily used or converted. Also, the IPF format is best suited for Amiga floppies since it's there it's used the most.

How many OS X apps do you know that can handle the IPF format? How many PC applications?
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter

Darkstar
User avatar
Donator
Posts: 918
Joined: Fri May 14, 2010 1:29 pm
Location: Southern Germany

Re: Authenticating releases

Post by Darkstar »

True, not many emulators support it yet (but more than 4 ;) ).
The advantages of Kryoflux are that you don't have to keep the old hardware around. A laptop with an old floppy drive is enough to dump a disk for *every* system. Converting that into any other disc based format is quite easy, all that's needed is an MFM decoder (or FM, or GCR, or whatever format the disk is in) and a few of them are already being developed (and some formats can already be generated from IPF with the dtc.exe from Kryoflux)

The copy protection problem though is virtually unsolvable without more emus supporting the IPF format (or any equivalent format)

Otherwise you will end up with either hacked versions of the disks or non-working images, both of which are equally bad


-Darkstar
I upload stuff to archive.org from time to time. See here for everything that doesn't fit BA

PeterPanPirate
User avatar
FTP Access
Posts: 290
Joined: Fri Jun 17, 2011 9:19 pm
Location: In the Bios

Re: Authenticating releases

Post by PeterPanPirate »

Darkstar wrote: The copy protection problem though is virtually unsolvable without more emus supporting the IPF format (or any equivalent format)
-Darkstar
Doubt many will. Can't ee VMware, Virtualbox or Bosch suppoting it anytime soon, and DOSBox already said they won't so the IBM IPF's is already dead. Unfortunately IPF is heading the Same root as td0, as beyond The Amiga it's not used actively. There is for the ZX/ST but it's a mission to use on the ST and can't remember a emu for the ZX. The one thing that's killing the format, is the Devs atm, and their licence they have in place, and you need to send them your files to make them ipf's.
Windows 8 on mobile devices and tablets is akin to Dr Jekyll: a tortured soul hoping for redemption, On a regular PC, Windows 8 is Mr Hyde: a monster that terrorises poor office workers and strangles their productivity.

Darkstar
User avatar
Donator
Posts: 918
Joined: Fri May 14, 2010 1:29 pm
Location: Southern Germany

Re: Authenticating releases

Post by Darkstar »

The problem is that there's no other equivalent low-level format in existence. So maybe it won't be IPF but there'll definitely be something similar at some point. The fact that some projects say they won't use it is also not set in stone and might change, for example when someone comes around and re-implements the IPF decoder as a GPL library (like libdisk for example, which is not GPL though)

As for VMware, VirtualBox and Bochs, those are not emulators from any "preservation" point of view :-)

-Darkstar
I upload stuff to archive.org from time to time. See here for everything that doesn't fit BA

P4R4D0X
User avatar
FTP Access
Posts: 128
Joined: Mon Jul 25, 2011 3:12 pm

Re: Authenticating releases

Post by P4R4D0X »

The IPF format is great but the choice of license pretty much is bad. That's why only a few emulators support it. If Softpres makes the format open then it would be popular. They better relicense it to a more compatible license for projects like GPL. Clearly they are doing this to cover up their asses and to sell more KryoFlux units. People submit RAW steam data to Softpres and they convert the disks to a .ipf format. In theory this is good, but they don't make it public, except to those who dump disks. And that's why CAPSDI was born.

IPF support won't be implemented in DOSBox because the licensing doesn't suit them. Also stuff should be implemented in order to work.

http://www.vogons.org/viewtopic.php?f=32&t=34332
http://www.vogons.org/viewtopic.php?f=3 ... 32#p304752
Regarding the IPF, the first issue with implementing it is that DOSBox does not fully, if at all, emulate the NEC 765 Floppy Controller. DOSBox instead emulates Int 13h floppy writes and DOS calls. The copy protection schemes which the IPF seeks to preserve rely on the behavior of that controller. So there is the first obstacle.
http://forum.kryoflux.com/viewtopic.php?f=4&t=584#p5125
I contacted one of the maintainers of DOSBox ages ago. The answer was something like "unless this is free (GPL), get lost". Now the decoder is free under a modified MAME licence. I think it still does not fit for them... It could also be that the code is still in a state where low level floppy support - which would be needed for correct IPF interpretation, is missing.

There's no reason to convert IPF to another format, at least not if you want to keep everything IPF stands for. Otherwise a solution to access and extract the data, like a plugin for Total Commander would make more sense.
In the current state of the IPF, I'm against it. The Software Preservation should open up the format or it will be forgotten quickly. That way preservation of copy protected disks would make sense, also more stuff could be saved before disk degradation occurs.

Darkstar
User avatar
Donator
Posts: 918
Joined: Fri May 14, 2010 1:29 pm
Location: Southern Germany

Re: Authenticating releases

Post by Darkstar »

Their decoder is not GPL. The IPF format itself is quite well documented (for a "closed format") and has at least one Public Domain implementation (which I linked to in a previous post).Since for most emulators you don't need their WD1772 implementation but only a sector-level reconstruction (at least as a starting point and alternative to regular sector-based formats) that's all you need. IPF is not really that difficult to parse anyway, as I said, the only thing you need is an MFM decoder which can be written in ~20 lines of code. Many copy protections can also be emulated on a sector level (e.g. unformatted tracks), there's often no need for the lowlevel WD1772 emulation.

I don't like SoftPres' choice of license either, but it's their choice, and if the MAME license were really that bad, MAME/MESS itself would have been "forgotten quickly" by now too ;-)

-Darkstar
I upload stuff to archive.org from time to time. See here for everything that doesn't fit BA

mrpijey
User avatar
Administrator
Posts: 8178
Joined: Tue Feb 12, 2008 5:28 pm
Contact:

Re: Authenticating releases

Post by mrpijey »

Either way, the only way IPF would be good for us would be if:
  • Easy to create. No special hardware or software, just a command line tool or simple GUI to create the image from original floppy.
  • Good software support. Must be easy to convert from IPF to IMG or whatever an emulator uses.
  • Cross platform tools. You must be able to handle the image regardless if you use Windows, OS X or Linux.
Despite IPFs design to preserve everything it doesn't furfill any of these requirements. And as far as I know, no advanced image format meets these requirements as they all are very proprietary, locked down or simply generally unused. And that's a shame really.

But hey, if someone can make a simple dumper tool that works on regular PC floppy drives (USB ones especially) and at the same time make a converter tool then yes, I would definately consider the format. if not then it's just like td0 and the others - just a little obscure format that is difficult to handle. The number one problem with all things being digital is to preserve it in a format that is well supported for a long time, we can't convert the whole collection to a new format every so many years. So perhaps we have to live with the fact that copy protections etc get lost in the tides of history? I try to use CloneCD/Alcohol120% when I dump optical discs but for floppies all I got is raw img basically.

Addition: For the sake of preservation I did order myself two KryoFlux units, it seems to be a good piece of hardware to use for preserving the original floppies, and it saves me the need of using a separate PC, Amiga and Mac to save the floppies. I've also contacted the devs and asked if it's possible to develop a conversion tool to convert to img etc (even if copy protections etc are lost) so we can easily use it in emulators and such.

Another addition: I've read up on the KryoFlux hardware and the IPF format and despite its technical advantages the entire thing is doomed to fail. Why? Because the developers do absolutely nothing to open up the format and hardware to the public, despite their claims of trying to preserve things. The KryoFlux hardware creates a "stream" of the floppy that has to be sent to them so they can convert it to IPF. So you can't do it on your own, which means you need to rely on them to do it, and as soon as they decide that they are tired of it the format along with everything else disappears. Also, the IPF format is not supported by anything significant (DOSBox for example will never support it) and only a very select few emulators support it. I doubt very much any decent PC emulator will ever support it, PCE does but it's far away from being an emulator to count on. So no support and tight control of the hardware... doomed to fail.
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Channels: Discord :: Twitter

Darkstar
User avatar
Donator
Posts: 918
Joined: Fri May 14, 2010 1:29 pm
Location: Southern Germany

Re: Authenticating releases

Post by Darkstar »

mrpijey wrote:
  • Good software support. Must be easy to convert from IPF to IMG or whatever an emulator uses.
  • Cross platform tools. You must be able to handle the image regardless if you use Windows, OS X or Linux.
Give it some time, I'm sure some people will perfect libdisk or create something similar. There's (sadly) not many people interested/involved in preservation work / coding.
mrpijey wrote: But hey, if someone can make a simple dumper tool that works on regular PC floppy drives (USB ones especially) and at the same time make a converter tool then yes, I would definately consider the format.
There's no way you can use USB floppies for dumping non-trivial disks. Most of these don't even work with the special install floppies used in Windows 95 or OS/2 (formatted at 1.7MB). They have all the data decoding built-in. What you want/need is raw bitstream timings. Even original PC floppy controllers can only do that in a very limited fashion (and these tend to get rare, since all the new "integrated chipset" peripherals start stripping away most of the complicated stuff and present themselves more and more like USB floppies)
mrpijey wrote: we can't convert the whole collection to a new format every so many years.
It's a nondestructive operation (that could be automated given the right tools, see above) so I don't see a problem.
But I wasn't suggesting moving to IPF as primary format. That would be overkill for most disks, and since many(?) of the disk images on the FTP are probably not from "original" disks but from copies (of copies..) there's nothing that would be gained. BUT for all disks with copy protection (or bad sectors), IPF would be vastly more helpful than no image file at all (or one with missing sectors) since you can recover bad blocks from an IPF file quite easily (it's in fact one of the things SoftPres/KryoFlux will do for you when you send in your dumps)
mrpijey wrote: So perhaps we have to live with the fact that copy protections etc get lost in the tides of history? I try to use CloneCD/Alcohol120% when I dump optical discs but for floppies all I got is raw img basically.
Yeah, and even the Alcohol120% images don't include all data required for 100% perfect replication of copy protection schemes (e.g. starforce, PSX libcrypt, etc.). They include their own virtual CD driver so that they can later "fake" the copy protection. When you think about it, that is even worse: What if the company behind Alcohol120% goes bankrupt? Their virtual CD drive is also closed source, and you end up with an image that you *know* is protected by some protection scheme, but with no idea on how to fake/emulate that protection... All your protected ISOs are suddenly worthless
(yeah, I know I'm exaggerating a bit, since the old versions will still run on your PC and all, but I hope you see what I mean)
mrpijey wrote: Another addition: I've read up on the KryoFlux hardware and the IPF format and despite its technical advantages the entire thing is doomed to fail. Why? Because the developers do absolutely nothing to open up the format and hardware to the public, despite their claims of trying to preserve things.
Not exactly true, they did open up their decoder library (I regard the MAME license as open, even if it's not OSI certified) and they are still open to the idea of releasing even more as "open source" in the future. At this point they're only concerned about everyone making IPFs of all their cracked disks so that later finding the "correct" dumps is as hard as it already is today. Right now you can be pretty sure that an IPF file comes indeed from an original, unmodified disk. I don't think that's such a bad course of action.

The right thing for them to do, obviously, would be to cryptographically sign all their IPF files and then release the format in a completely OSI certified way, which would solve their main concern. As far as I can tell that is still something they're considering
mrpijey wrote: The KryoFlux hardware creates a "stream" of the floppy that has to be sent to them so they can convert it to IPF. So you can't do it on your own, which means you need to rely on them to do it, and as soon as they decide that they are tired of it the format along with everything else disappears.
You seem to think that the intermediate file format that gets created from the device is somehow encrypted or proprietary. That is simply not the case. The bitstream format that the KryoFlux device sends to the PC (I think it's called DRAFT) is very simple and doesn't really *need* any documentation (other than maybe the timebase used, and even that can be guessed from the raw data). It basically contains the number of "timer ticks" between two opposing magentic areas on the floppy (try opening one of the intermediate files in an Hex Editor that can generate histograms, you'll see what I mean). It's the lowest common denominator from which all other formats can be easily re-created, not only now but in the future as well.

Of course, right now, you need to send that file (it's actually a bunch of files, totalling many megabytes for a single floppy) to KryoFlux to have it converted to IPF. But you don't need to do that. Maybe someone will come up with their own GPL'ed "raw" file format. Boom, instantly converted. Or KryoFlux decides to open up the format and release an IPF encoder. Boom, instantly converted. Or you decide it's enough to simply have an IMG file.. boom, instantly converted.

Finally, we all know that the conclusion "it's not open, so it will fail" does not hold true :) Look at Apple's success. And Microsoft. etc. etc

-Darkstar
I upload stuff to archive.org from time to time. See here for everything that doesn't fit BA

Post Reply