BetaArchive Logo
Navigation Home Database Screenshots Gallery Image Uploader Server Info FTP Servers Wiki Forum RSS Feed Rules Please Donate
UP: 36d, 1h, 37m | CPU: 13% | MEM: 5897MB of 12287MB used
{The community for beta collectors}

Post new topic Reply to topic  [ 28 posts ]  Go to page Previous  1, 2
Author Message
 PostPost subject: Re: Authenticating releases        Posted: Fri Jul 12, 2013 11:01 pm 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
7591
Darkstar wrote:
Give it some time, I'm sure some people will perfect libdisk or create something similar. There's (sadly) not many people interested/involved in preservation work / coding.


Yeah, well while we wait floppies die the slow death :).

Darkstar wrote:
There's no way you can use USB floppies for dumping non-trivial disks. Most of these don't even work with the special install floppies used in Windows 95 or OS/2 (formatted at 1.7MB). They have all the data decoding built-in. What you want/need is raw bitstream timings. Even original PC floppy controllers can only do that in a very limited fashion (and these tend to get rare, since all the new "integrated chipset" peripherals start stripping away most of the complicated stuff and present themselves more and more like USB floppies)

Works very well actually, but of course it depends on the USB floppy. The one I got (an expensive Sony one) actually does proper bitstream dumps so I have no issues dumping odd formats, I can even read out Amiga, Mac and Atari floppies with special software. But most USB drives won't work of course as they are cheap and uses internal decoding.

Darkstar wrote:
It's a nondestructive operation (that could be automated given the right tools, see above) so I don't see a problem.
But I wasn't suggesting moving to IPF as primary format. That would be overkill for most disks, and since many(?) of the disk images on the FTP are probably not from "original" disks but from copies (of copies..) there's nothing that would be gained. BUT for all disks with copy protection (or bad sectors), IPF would be vastly more helpful than no image file at all (or one with missing sectors) since you can recover bad blocks from an IPF file quite easily (it's in fact one of the things SoftPres/KryoFlux will do for you when you send in your dumps)

Well, it is a problem because it takes a lot of time and our collection grows by the day. And I sure don't want to go through every floppy image here and convert it... Also, IPF is bad for that very reason, that you need a third party to create the images. One day they may decide they had enough and boom, format is dead. And they have admitted that they absolutely not want to release the format for free. So, we got an another proprietary format, just like Teledisk and others. Good for the developers of the format, bad for everyone else. So despite the lack of custom sector support and protection etc img is still the best format because everything support it without frills.

Darkstar wrote:
Yeah, and even the Alcohol120% images don't include all data required for 100% perfect replication of copy protection schemes (e.g. starforce, PSX libcrypt, etc.). They include their own virtual CD driver so that they can later "fake" the copy protection. When you think about it, that is even worse: What if the company behind Alcohol120% goes bankrupt? Their virtual CD drive is also closed source, and you end up with an image that you *know* is protected by some protection scheme, but with no idea on how to fake/emulate that protection... All your protected ISOs are suddenly worthless
(yeah, I know I'm exaggerating a bit, since the old versions will still run on your PC and all, but I hope you see what I mean)


Enough software actually supports the MDF/IMG format Alcohol supports, so it's not the same issue like with IPF or Teledisk where _one_ application (itself) supports it.

Darkstar wrote:
Not exactly true, they did open up their decoder library (I regard the MAME license as open, even if it's not OSI certified) and they are still open to the idea of releasing even more as "open source" in the future. At this point they're only concerned about everyone making IPFs of all their cracked disks so that later finding the "correct" dumps is as hard as it already is today. Right now you can be pretty sure that an IPF file comes indeed from an original, unmodified disk. I don't think that's such a bad course of action.


Well i read through their forums and they are not interested in opening up the format really. If they were they would have done so by now as well since the format got some traction with the Amiga SPS releases etc. Also they can't be sure it comes from a perfect disk because they convert the stuff people send in. It's not like you need to send the actual floppy... so I could easily change a few bits of a floppy, dump it and call it "original". All they can do is maintain a checksum database of the image/files and then compare it when other releases drop in. And they can do that with the format being open as well so nothing really changes, except that if they make it open then no one will publish everything to them and they won't be able to get a large collection for free. And that they made the decoder libs open means nothing as there's no way to ENCODE the IPF files, only read out stuff from them. So loss either way.

Darkstar wrote:
The right thing for them to do, obviously, would be to cryptographically sign all their IPF files and then release the format in a completely OSI certified way, which would solve their main concern. As far as I can tell that is still something they're considering


Well, a signature of some kind would be good, i.e something they sign so people can see the source, i.e official from the SPS group or elsewhere. We could do the same by signing it with a BA crypto signature etc. Still wouldn't stop people from recreating the floppy and redump it with their own sig... either way the community loses when things are kept closed.

Darkstar wrote:
You seem to think that the intermediate file format that gets created from the device is somehow encrypted or proprietary. That is simply not the case. The bitstream format that the KryoFlux device sends to the PC (I think it's called DRAFT) is very simple and doesn't really *need* any documentation (other than maybe the timebase used, and even that can be guessed from the raw data). It basically contains the number of "timer ticks" between two opposing magentic areas on the floppy (try opening one of the intermediate files in an Hex Editor that can generate histograms, you'll see what I mean). It's the lowest common denominator from which all other formats can be easily re-created, not only now but in the future as well.


And yet no one has made anything available to convert it to the format of your choosing. Why? Because the format has undocumented stuff in it and it's not shared... so it's not as easy as you think. For the streams to be properly encoded into a usable format it has to be 100% documented and supported. And they simply don't want to since for the time being they want the absolute sole rights of handling the streams and make them into IPF, a format which is also controlled by them.

Darkstar wrote:
Of course, right now, you need to send that file (it's actually a bunch of files, totalling many megabytes for a single floppy) to KryoFlux to have it converted to IPF. But you don't need to do that. Maybe someone will come up with their own GPL'ed "raw" file format. Boom, instantly converted. Or KryoFlux decides to open up the format and release an IPF encoder. Boom, instantly converted. Or you decide it's enough to simply have an IMG file.. boom, instantly converted.


Darkstar wrote:
Finally, we all know that the conclusion "it's not open, so it will fail" does not hold true :) Look at Apple's success. And Microsoft. etc. etc

-Darkstar


Well, "maybe" someone will. "Maybe", "perhaps", "in the future"... all words telling something that is not set in stone. Which is the problem, we need something NOW, not in 5-10 years or even longer... there's been so many truly "lossless" image formats, and none has been truly open and supported.

Also, I never said that because it's not open it will fail, but it's most likely as no one other than themselves will use it. And mentioning Apple and Microsoft isn't really a good example as they are VERY large (thus don't need support by others as the majority already uses them) and also not alone (as there are alternatives). We don't have many alternatives to a truly "lossless" imaging format for floppies.

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Tools: Alcohol120% (Portable)
Listings: BetaArchive Database (beta)
Channels: Discord :: Twitter


Top  Profile  WWW
 PostPost subject: Re: Authenticating releases        Posted: Sat Jul 13, 2013 12:03 am 
Reply with quote
Donator
User avatar
Offline

Joined
Fri May 14, 2010 1:29 pm

Posts
828

Location
Southern Germany

Favourite OS
IRIX 5.3
mrpijey wrote:
Yeah, well while we wait floppies die the slow death :).

true :( :(

mrpijey wrote:
Works very well actually, but of course it depends on the USB floppy. The one I got (an expensive Sony one) actually does proper bitstream dumps so I have no issues dumping odd formats, I can even read out Amiga, Mac and Atari floppies with special software. But most USB drives won't work of course as they are cheap and uses internal decoding.

Wow, I never had one of these. I tried a few but all were crap (even the brand ones, Dell etc.)

mrpijey wrote:
Enough software actually supports the MDF/IMG format Alcohol supports, so it's not the same issue like with IPF or Teledisk where _one_ application (itself) supports it.

Yes but do they also support all the protection schemes that Alcohol's Virtual CD drive supports? DaemonTools supports most of them (although I admit I haven't tested every single one), but are there any others?

mrpijey wrote:
so I could easily change a few bits of a floppy, dump it and call it "original".

You could do this, but only after the disk has been dumped (and even then it would be difficult to hide because of the statistic deviations you'd have to fake). If you tried on the physical disk it would be easily detectable since writing a single sector inevitably changes the sync marks on the track slightly, which can be detected on the dumped data stream. That's actually how they find out if any disks have saved highscores or something on them. Tracks written by the Tracer are much more "perfect" than anything you can write with your own PC. An interesting read are their WIP pages spanning about a decade and offering many insights on lowlevel disk formats

mrpijey wrote:
And yet no one has made anything available to convert it to the format of your choosing.


libdisk contains an IPF parser and can (currently) convert ADF files. Adding support for IMG is rather straightforward (and I think he already did that, for testing, but I cannot seem to find the link right now):

MESS has an IPF parser and their floptool can convert from IPF (and maybe even to IPF but I haven't tested that)

Code:
d:\Devel\mess>floptool
floptool - Generic floppy image manipulation tool for use with MESS

Usage:
       floptool.exe identify <inputfile> [<inputfile> ...]
       floptool.exe convert [input_format|auto] output_format <inputfile> <outputfile>

Supported formats:

            mfi - MESS floppy image [mfi]
            dfi - DiscFerret flux dump format [dfi]
            ipf - SPS floppy disk image [ipf]
            mfm - HxCFloppyEmulator floppy disk image [mfm]
            adf - Amiga ADF floppy disk image [adf]
             st - Atari ST floppy disk image [st]
            msa - Atari MSA floppy disk image [msa]
          pasti - Atari PASTI floppy disk image [stx]
            dsk - CPC DSK Format [dsk]
            d88 - D88 disk image [d77,d88,1dd]
             pc - PC floppy disk image [dsk,ima,img,ufi,360]
           dc42 - DiskCopy 4.2 image [dc42]
      a2_16sect - Apple II 16-sector dsk image [dsk]
      a2_rwts18 - Apple II RWTS18-type Image [rti]

Example usage:
        floptool.exe identify image.dsk


mrpijey wrote:
Why? Because the format has undocumented stuff in it and it's not shared...

The whole format is documented in the header files of the source code (e.g. in CAPSImage\include\caps\*.h). While you cannot (obviously) use that header file directly in a GPL project, there's noone stopping anybody from writing a clean-room spec from the structure definitions and reimplementing it as GPL code (or even as PD code, like Keirf did)
Also, here is the full file format specification of the STREAM format

mrpijey wrote:
And they simply don't want to since for the time being they want the absolute sole rights of handling the streams and make them into IPF

Again, I never said you should convert all your IMGs to IPF. Au contraire. Thus we don't *need* to create IPFs ourselves. Everyone who has a KryoFlux board can right now dump any disk he has and send the dumps to SoftPres to receive the IPF file. It's like converting yourself, only a bit slower. They make sure that your dump is good, fix any broken tracks if they have the same image from another source, and you get a perfect IPF file. Yes, it's a closed format, but you still get to keep the raw stream files (they compress quite nicely ;) ) so you don't even lose anything. So if I went to all that trouble (buying a KryoFlux, waiting a few days for my IPF file to arrive), why should I only upload an inferior copy of the disk (which might not even run unpatched if it's protected) to BA instead of the better (i.e. more perfect) IPF file? Maybe in addition to the IMG file, I don't care. But if it's a genuine dump from an original disk, I'd much prefer an IPF file even if I cannot (yet) use it directly with DOSbox. Some of the stuff here is so rare that it makes it all the more important.

mrpijey wrote:
Also, I never said that because it's not open it will fail...

Actually you did ;-)
mrpijey wrote:
...the entire thing is doomed to fail. Why? Because the developers do absolutely nothing to open up the format and hardware to the public


Anyway, this thread is diverting more and more from the original topic, I think if we wanted to discuss this any further it would be best to create a new thread specifically for this discussion. I don't want to hijack the thread too much ;-)

-Darkstar

*Edit: added link to STREAM format

_________________
I upload stuff to archive.org from time to time. See here for everything that doesn't fit BA


Top  Profile
 PostPost subject: Re: Authenticating releases        Posted: Sat Jul 13, 2013 1:23 am 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
7591
Thread is continued here.

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Tools: Alcohol120% (Portable)
Listings: BetaArchive Database (beta)
Channels: Discord :: Twitter


Top  Profile  WWW
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 28 posts ]  Go to page Previous  1, 2




Who is online

Users browsing this forum: No registered users and 8 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  

All views expressed in these forums are those of the author and do not necessarily represent the views of the BetaArchive site owner.

Powered by phpBB® Forum Software © phpBB Group

Copyright © 2006-2019

 

Sitemap | XML | RSS