Floppy formats, emulation, etc.

[Battler]

Basically I'm starting this thread so we can discuss everything about floppies.

I'll start with a question of my own - what are the known floppy copy protections?

Also, what is the maximum number of tracks that is known to be possible to store on any of the three standard floppy sizes (8", 5.25", 3.5")? I know the standard numbers of tracks are 35/77, 40/80, and 80, respectively, but what is the maximum?

[computebrute]

Here are the types of floppy disk copy protection:

Intentional Disk Errors:

This protection consists of a disk error purposefully placed on a header, sector, or an entire track, then its existence is checked for by the protection. If the error isn't found, it assumes this is a copy. In the beginning, there were no copy programs except what came with the operating system, and it could not read or write these errors. Like most protections, though, it only foiled casual copiers. Early, clever hackers figured out how to scan an original disk for these and the methods for recreating the errors were not difficult. Soon after, programs came out that could effortlessly detect and reproduce these errors which marked the end of this era.

Extra Tracks:

Most floppy drives and media are rated to use 40 tracks on a 5.25" disk, and 80 tracks on a 3.5" disk, however most hardware will "step out" a bit further and data is stored here and checked.

"Signatures" in the Header, Sector, or Tail gaps

As mentioned in the copier section, when a disk is copied with either a fast copier or a nibbler, the gap data is not copied directly. Gaps are "inert" bytes, meaning they aren't normally used as data, but just a space filler that is ignored by DOS. There are some protections that check for specific bytes here, or even the length of the gaps, and know they're a copy if it doesn't match

Alternate Density

Some protection methods take advantage of the fact that disks can only be written at certain densities by a consumer disk drive. Because original disks are not written with consumer drives, they can be written at a different density that can still be read properly by a consumer drive. A copy program may read the data at the standard density only, resulting in corrupt data.

Half Tracks/Quarter Tracks

On a 40-track drive, it is usually possible to read from the tracks in between the normal tracks, so companies would master data here. This foiled many copiers because they didn't search for and copy these "extra" tracks by default, so they instead got garbled, invalid mixes of the "real" data when trying to read the tracks as if they were normal. Copiers later came out that could scan for and detect these and copy them properly.

FAT Tracks

Disks with this protection are mastered with an 80-track drive, but the drive used with the consumer hardware is only 40-tracks. Because of this, the drive will be able to see different data in these "half tracks" that it cannot write without destroying data on adjacent tracks

Track Skew/Timing Checks

This protection involves placing a track on the disk that is a specific skew offset from the one before it, or sectors that are a specific distance apart on the track. The protection is checked by timing these and verifying it is what is expected from the original disk.

Long Sectors

As mentioned, the drive may not be able to fit an entire track in RAM, so it must be broken down into parts and stitched together on the destination disk. If you make a sector longer than will fit in RAM, it can't be copied with a normal drive.

Custom Formats

These protections entail changing the way that the disk encoding is interpreted in a custom DOS. If you write your own DOS, you can use your own disk format entirely, ignoring common sync, gap, header, and data conventions altogether and use whatever you want. You only have to adhere to the clocking limitations, but even that didn't stop later innovations.

Long or Short Tracks

Normal drives spin at 300 or 360 RPM and can "write" data at a certain density. However, the drive can "read" more than that, within some margin of error. Some protections take advantage of this and use a different bit rate to store more (or less) data on the track than could be written back out at the normal speed. Some copiers would "trim" the data in as non-destructive way as possible (reducing inert data) and it was also possible to slow or speed up the disk drive motor to foil this in some cases, but that is inconvenient.

Encoding "Violations"

Since disks are encoded using certain rules (GCR, FM, MFM) by consumer floppy drives and their firmware, they are not typically able to write data outside these rules. Duplicators can write disks outside these rules that exploit how the disk is decoded, which can be detected as a protection.

"Weak Bits"

When a drive sees no flux transitions within a certain time, it can read as random data. Most nibblers can't duplicate this, so protections checked data in these areas on the disk somewhere to see if the data changes.

"Fuzzy Bits"

If a flux transition is encoded right in between two different timings, it can be read one way on one revolution, and another way the next, causing the data to fluctuate or "fuzzyness". The effect is similar to "weak bits".

"Strong Bits"

Some duplicators can write a stronger than normal flux pulse on a disk, which can be detected by some controllers as they overload the read circuitry.

Signature (Key) Tracks

This protection revolves around a track being mastered in a non-standard way. It can be non-formatted, filled completely with a repeating sequence, or a combination. These tracks generally foil most all copy programs because they either think they are blank, cannot handle the long sequences, or they are just read incorrectly.

No SYNC

Some copy protection used tracks or sectors on the disk that had no sync pattern. Most copiers cannot read this data and just detect them as empty.

"SpiraDisc"

This protection name hails from the Apple ][, but variants appeared on other platforms. The data is loaded by stepping a half or quarter track at a time. If any deviation in time from where it expects the data to be and when is found, or the data stream is detected as corrupt, it fails. Consumer drives cannot duplicate this without special routines that would essentially be recreating the data.

Physical Damage, or "Laser Holes"

This protection consists of physical damage to the disk media itself. A damaged disk cannot be written to, and will result in errors, so the protection will check for this by writing a track or sector and verifying if it is good (which would fail). These can be tiny, barely visible "laser holes" or have been seen as obvious scorch/burn marks.

[RentedMule]

One job I had, I was responsible for Mitchell on Demand installations for a garage. They used a key diskette, which had only 39 tracks. The 40th track's data was marked as bad sectors in the FAT table, but if you physically tried to read/write from/to them, it would fail. You couldn't format these things for 40 tracks either, or it would fail.

Fortunately, we found a nice workaround. Write protect the actual key diskette, put it in the drive while installing. It would fail when trying to update the disks data that you were one key shorter, but you could then put in a standard floppy into the drive that wasn't write protected, and it would write to it instead, completing the installation.

[JustZisGuy]

Typical 1.44mb drives can usually access up to 83 tracks. A few (mainly Toshibas, if I recall correctly) can reliably access 84.
At least some 1.2mb drives can access 83 tracks.
And 360k drives usually can access 42.

Of course, some drives can be physically limited to the standard 40/80 or limited by "smart" electronics.

Using extra tracks for copy protection was a tad risky, as the above or BIOS limits in some clones could prevent access, or even risk damaging the drive.

[Battler]

Thanks for the information, guys. Now I have a question - what is the minimum number of tracks for each floppy format? I heard of 35-track media but that's allegegly 8" only, and RentedMudle mentioned 39-track floppies. For high-density floppies, I've heard of 77-track floppies (with 8 1024-byte sectors, a format going all the way back to 8", but in Japan was used for 5.25" and 3.5" HD floppies too). But what is the actual minimum?

[JustZisGuy]

Well, you can format a disk with as little as one track, but I assume you mean the lower physical media/drive limits. Very early 5.25" 48tpi drives and disks could only use 35 tracks. The Apple II used 35 track 5.25" drives, and that limitation stayed with it its entire life. Some Micropolis 100tpi (instead of the normal 96tpi) 5.25" drives were limited to 77 tracks.

Off hand, I don't know of any 3.5" drives that supported less than 80 tracks, but some systems may have only used 35/40/77 due to ROM, controller, or OS limitation.

There is some good info info about oddball drives here: http://www.retrotechnology.com/herbs_stuff/drive.html

Actually, they do mention an odd Epson 3.5" drive that used 67.5tpi with 40 tracks per side.

[tarlabnor]

Would be also good to create a table with information: which floppy images can (or can not) be:
1. succesfully mounted under sertain emulators
2. can be succesfully "written" with original software under emulators onto virtual floppy

Example:

emulator: PCem-X
supported image formats: *.img, *.ima, *.raw, *.144, *fdi, *pef, *flp
emulated fdc: intel 82078
selected drive mode: 3.5" 1.25/1.44 HD (3 mode)

format 1: *.td0
used soft: teledisk 2.15 (direct mode)
result: success

format 2: *.ddi
used soft: diskdupe 7.0
result: success

format 3: *cfd
used soft: pu_1700 + pu_copyfd
result: success

format 4: *.cp2
used soft: copyiipc 6.0 + snatchit
result: fail

etc.

[Battler]

- tarlabnor: That would indeed be a good idea. However, it would be good to also specify version or, in case of PCem-X, date of the binary, because the FDC code is subject to change (massively so by my next release - working on implementing support for actual FM/MFM-encoded bit streams).

[tarlabnor]

However, it would be good to also specify version or, in case of PCem-X, date of the binary, because the FDC code is subject to change

It's latest (I guess) available version - 20150219.

I'm not sure that it's possible to check every combination of emulated FDC/OS/FD imaging software, but some kind of DB with succesful/unsuccesful attempts is definitely possible.

massively so by my next release - working on implementing support for actual FM/MFM-encoded bit streams

Great. I wonder if it could help to emulate some formats (f.i. *.tc - TransCopy) that support copy protections. Atm the only known emulator that can do it is PCE.

[Battler]

- tarlabnor: The thing is, none of those formats is adequately documented. However, I will add support for IPF via the CAPS library. That's of course in addition to my own .PEF format and regular .IMG and .FDI (the Japanese FDI format, not the Amiga FDI format stock PCem now supports).

[x010]

Have no idea about floppy stuff , but if you were to use an ISO tool to convert a bootable floppy image to an exact(including the boot sector) ISO , would that be possible?

And , what is the theoretical speed of a floppy disk connector? Limited to the IDE connector , right?

Finally , why is it so if one were to format a wrong density on a floppy , then it wouldn't work properly?

[Darkstar]

by the way, I hope you are also submitting all your changes upstream to Tom? And pull in his changes from time to time?

[Battler]

by the way, I hope you are also submitting all your changes upstream to Tom?

I do provide my source code. Then it's his choice what of the changes he wants to take to upstream, and which doesn't. And it seems his reaction to my fork has been to start ignoring me anyway.

Also, most of the changes he wouldn't accept anyway. The ability to set back the clock he has outright rejected on Vogons where he clearly stated that to him, the guest machine always having the host's time is not a bug therefore he won't do anything about it. The Ne2000 he claims is too incomplete for him to accept. My JEGA emulation is right now based on plenty of guesswork based on what few software I have for the AX (such as DOS 3.21, and the DOSSHELL .VID file) and what I am able to find on the Internet (which is not much). My CPU-related changes he won't accept because he clearly said he doesn't accept CPU changes. And finally, my FDC-related changes are incompatible with his own new FDC code that supports the Amiga FDI format as well as raw images.

And pull in his changes from time to time?

I do do that, of course.

[Darkstar]

Sorry to hear that, I thought he was pretty cooperative, but it seems I was mistaken. You don't update your GitHub Repo anymore, right? So the only way to get the sources are the ZIPs on SoftHistory?

[Battler]

- Darkstar: Yeah, the only way to get up-to-date sources are the ZIP's on SoftHistory.

[anormal]

battler: the first think i'd do if you are interested about copy protections is reading the document in drCoolZic page, it's based aroud Atari ST copy protections, but it's the best thing you'll found in the net, afaik.

for me, supporting copy protections in image floppys need flux information to be recorded, RAW (and the compressed or processed...) but you can't do it with a simple sector format and some headers. As you know, PCE supports psi/pfi/pri formats, and KF outputs raw flux streams and CT Raw, also Super Card uses it's format.

If i were to support IBM PC floppies protection in my emu i'll do, by order of importance

KF & SCP formats: raw and compressed formats
(at this point i'll ignore IPF format, because Softpress foundation has released just a pair of IBM PC games in this format, in the possible future they release more games in this format, is very easy to add IPF support)
PCE formats: as you know they are superset of minor formats
---- until here you got sure you don't need any more information to support correct format emulation
---- but if you want minor formats..., with less information, losing copy protection information and forcing hacks
---- in emulation, then add:
Transcopy format: this is interesting, but knowing the number of TC out there...
Anadisk, Teledisk, snatchit, samdisk: nice formats with some good information inside (support some protection)
DDI format
Img,vfd,imz: simple sector formats, copy protection information is lost, this will not run in good emulation

(Maybe someone with good knowledge of KF/SCP/TC, Hampa?, could comment more about this list)

What information support your PEF format?, does it support multiple lectures by track?, flux information?
-------
About tracks... maybe anyone could confirm what i think: floppy surface is magnetical, no hard-tracks are defined
in the surface, so you could try to format a low density floppy in a high density drive, depending on quality of the magnetic surface you'll get all the tracks or you get errors,

Some drives (not floppies) support more tracks (42 or 83) and other drives support less tracks, you need to login in SCP and KF forums and read some threads there. Even the same model of the same drive could support more o less tracks, depending in calibration, etc...

KF and SCP software supports checking the drive with a floppy inside to check "how-far" your drive could get tracks, my Epson 5 1/4 drive is able to go to 42 tracks tracks for example

If you need more info just ask!
regards!





regards!

[Darkstar]

battler: the first think i'd do if you are interested about copy protections is reading the document in drCoolZic page, it's based aroud Atari ST copy protections, but it's the best thing you'll found in the net, afaik.

The WIP pages on the softpres homepage are also very detailed about various copy protection schemes.

[Battler]

- anormal: At this point, I have decided I will to the following:
1. Get my own code done, to support IMG/IMA/FLP/VFD (sector image), PEF (with various sub-formats - from sector image to track stream image), and FDI (Japanese FDI, not Amiga FDI) images;
2. Attempt to get the PCE code working in PCem-X as an option.

Once that is done, I can attempt, if I get to understand it well enough, to support TeleDisk too (but read only).

I wanted to implement IPF support via CAPS but now I read enough bad things about them that I have decided not to. They apparently don't give a damn about PC anyway, so why bother. And as for KryoFlux and such... maybe, one day...

What information support your PEF format?, does it support multiple lectures by track?, flux information?

The aim with the PEF format is to, in its most detailed subformat, support the most data a PC FDC can actually work with. I don't see the point of flux because I don't see why you need that. A MFM or FM encoded bit stream? Yes, of course. I am attempting to implement that (it's needed anyway for proper track streams since that way I can store missing clock bits, etc. properly). But the actual magnetic transitions - I just don't see what they bring that can not be achieved with a bit stream alone.
Also what do you mean by multiple lectures per track?

but you can't do it with a simple sector format and some headers.

Well for MFM or FM bit streams, that's indeed true. I just, at the time I thought of my header+sector image combination, didn't really think the clock bits were as important as I later found out they were (they're used, among other things, to sync the head to the sector!).

[mrpijey]

Considering we archive stuff here mainly in img, FDI, TD and KF formats these would be great to support. I don't care much about the IPF format as there's so very few tools that can actually make them, but KF Stream and CTRAW formats would be great to support as we can then directly use our dumped data in the emulator.

This endless conversion between formats is pointless and in the end we will lose information moving from one supported format to one less supported one. I see the valid points with your PEF formats, but it's just yet an another format that no one has heard of and that is basically used in a single emulator. Perhaps you should reach out to other emu devs and ask if they can support your format so we get more spreading of it, rather than constantly reinventing the wheel by introducing new formats (especially when there already exists good enough formats).

Just my personal opinion of course, but it's rather troublesome trying to set some standards in preservation when every emulator and tool out there decides to redo everything from scratch every time.

And the point with flux information? To store as much information as possible and to get as close as possible to the physical media as possible. In the end it may be 0's and 1's, but by storing the flux info you bridge the gap to the analog world and the floppy magnetic surface itself, storing the actual magnetic values created by the factory/duplicator which is far closer to a perfect preservation than just storing clock bits and whatnot. I am of course talking from a preservation point of view and not from practical emulation needs. But I still argue that the emulator should support these formats so we don't have to make yet an another conversion to a sub-optimal format. In the end you will probably have to make some tool to properly convert to your PEF format, and that time could be better spent by directly supporting the originating formats in the emulator, eliminating the need to convert data before it can be used.

We can't store a game or app in every conceivable format (img, raw, ctraw, ipf, stream, flp, vhd, pef, pce, fdi, tc and so on) so why not try to focus on saving it at the lowest level possible, which is a magnetic flux level. And if we then can get proper emus to support it then even better. Today we got KF, CopyIIPC and SCP which all works on a flux level. If you can support these formats then we got a perfect basis for an emulator that can directly work with the generated images. Or if you at least create a conversion tool which is included/embedded into the emulator that generates flawless images that works with every situation.

I know it's a difficult thing to do though, and everyone thinks their own format is superior to others, but remember how the computer world looked in the 80's? Everyone had their own formats, standards etc. It was a complete mess. How would the PC looked today if no one would have focused their development efforts into a single standard (IBM PC+clones)? I think we should at least on a small scale try to do the same here, focus on the most used formats, and support some low level ones.

Of course I am not a dev of your emu so I won't tell you what to do, just what I wished your emu would support. I think it's great we got a developing emulator that deals with common (and uncommon) PC formats, but no emulator is ever good if it doesn't support any decent standardized formats.

I got some development experience myself in drivers and I am contemplating at making a universal floppy emulator (much like Virtual Floppy Driver but for Win7+) and I've had some success testing some code, the only thing lacking is time. But this floppy emulator would only be useful if the emulators could tie directly to hardware (real or virtualized). The commercial virtualizers do this (VPC, VMWare Workstation etc), but will yours? Otherwise is rather pointless project...

Better so perhaps the efforts would be better put into making a generic library that can be used with emulators, one that supports all the various formats. So instead of every emulator having their own code for the floppy controller etc it could use an external library that also supported mounting of all common (and uncommon, pending on format documentation availability) formats. That way you don't need to focus on your own formats and just put the work into the emulator supporting the various other PC hardware... And it could then support your PEF format as well...

Much like the CAPS library, but more extensive...

Sorry for all the rant, but I am sitting here planning for the new BetaArchive FTP tool I am coding (which will standardize a lot on the BA FTP) and I thought of standards in terms of floppy image formats...

[Battler]

Considering we archive stuff here mainly in img, FDI, TD and KF formats these would be great to support. I don't care much about the IPF format as there's so very few tools that can actually make them, but KF Stream and CTRAW formats would be great to support as we can then directly use our dumped data in the emulator.

IMG and FDI (assuming you mean Japanese FDI) are already supported, and I even made PCem-X support IMG better than other emulators as I've gone to great length to make images of XDF floppies load correctly. TD and KF are sadly still both poorly documented.

This endless conversion between formats is pointless and in the end we will lose information moving from one supported format to one less supported one. I see the valid points with your PEF formats, but it's just yet an another format that no one has heard of and that is basically used in a single emulator. Perhaps you should reach out to other emu devs and ask if they can support your format so we get more spreading of it, rather than constantly reinventing the wheel by introducing new formats (especially when there already exists good enough formats).

I agree about that. Hence why in addition to PEF, I am also going to attempt to support PCE's formats too (hopefully, I can plug in PCE's FDC code into PCem-X without much hassle, but you never know what issues can there be).

Just my personal opinion of course, but it's rather troublesome trying to set some standards in preservation when every emulator and tool out there decides to redo everything from scratch every time.

Hence why the standards need to be done by the community. I am going to address this further in detail below.

And the point with flux information? To store as much information as possible and to get as close as possible to the physical media as possible. In the end it may be 0's and 1's, but by storing the flux info you bridge the gap to the analog world and the floppy magnetic surface itself, storing the actual magnetic values created by the factory/duplicator which is far closer to a perfect preservation than just storing clock bits and whatnot. I am of course talking from a preservation point of view and not from practical emulation needs. But I still argue that the emulator should support these formats so we don't have to make yet an another conversion to a sub-optimal format. In the end you will probably have to make some tool to properly convert to your PEF format, and that time could be better spent by directly supporting the originating formats in the emulator, eliminating the need to convert data before it can be used.

While I understand your point, you have to understand that already the 0 and 1 FM or MFM bit stream is hard to implement, a magnetic flux format is even harder to implement, even more so when it's in an undocumented (or poorly documented at best) format. I also think storing the magnetic transitions is pointless. The data on the floppy is by definition digital. The magnetic transitions, in the NRZi format in case of a PC floppy, are just a way to store digital bits onto magnetic media in a way that it can then be reasonably reliably read back. Considering that what a normal floppy drive reads from the floppy are in the end the 0's and 1's of the FM or MFM stream, in the end that is the absolute minimum that has to be preserved. Anything else adds no real useful information.
And please note I never said I'll never implement KF or CTRAW formats - I will when there's enough documentation available. And when the rest of the emulator matures to the point of being able to guarantee the correct timings for floppy emulation (right now, it by far can't, the CPU code is still too slow, and the entire thing is single-threaded so the CPU code ends up slowing down everything else, the emulated FDC included).

We can't store a game or app in every conceivable format (img, raw, ctraw, ipf, stream, flp, vhd, pef, pce, fdi, tc and so on) so why not try to focus on saving it at the lowest level possible, which is a magnetic flux level. And if we then can get proper emus to support it then even better. Today we got KF, CopyIIPC and SCP which all works on a flux level. If you can support these formats then we got a perfect basis for an emulator that can directly work with the generated images. Or if you at least create a conversion tool which is included/embedded into the emulator that generates flawless images that works with every situation.

Hence why we need to find a commonly agreed minimum that preserves every single bit from the floppy, while reasonably omits what is reasonably superfluous to make implementation easier. Yes, of course, having KF and CTRAW supported immediately would be perfect, but right now it's not the priority for me. One day in the future I am going to think about adding KF and CTRAW in, but right now, getting PEF to a reasonable degree of usability and implementing PCE's formats is as far as I can hope to get.

I know it's a difficult thing to do though, and everyone thinks their own format is superior to others, but remember how the computer world looked in the 80's? Everyone had their own formats, standards etc. It was a complete mess. How would the PC looked today if no one would have focused their development efforts into a single standard (IBM PC+clones)? I think we should at least on a small scale try to do the same here, focus on the most used formats, and support some low level ones.

The PC would have ended stagnating because of lack of competition. It is competition that drives the proponents of one format to improve theirs so it's better than the competing ones. Take that away and there's no more need to innovate as people will use whatever you make anyway, simply because there's going to be no other choice.
Of course some standardization is needed, for the sake of interoperability. But with most if not all emulator developers doing their work for free, and everyone having different priorities and ideas of what's the best format, it's going to be difficult.

Of course I am not a dev of your emu so I won't tell you what to do, just what I wished your emu would support. I think it's great we got a developing emulator that deals with common (and uncommon) PC formats, but no emulator is ever good if it doesn't support any decent standardized formats.

Well, hopefully I can implement PCE's formats and it can then lead to more people adopting them (which would in turn lead to even more emulators adopting them). But that's as far as I can hope to get in the near future. Anything further (KF and CTRAW) has to wait.

I got some development experience myself in drivers and I am contemplating at making a universal floppy emulator (much like Virtual Floppy Driver but for Win7+) and I've had some success testing some code, the only thing lacking is time. But this floppy emulator would only be useful if the emulators could tie directly to hardware (real or virtualized). The commercial virtualizers do this (VPC, VMWare Workstation etc), but will yours? Otherwise is rather pointless project...

If your floppy emulator provides an interface through which user-mode programs can talk to it, then I can easily make PCem-X talk to it. Also, Virtual Floppy Drive does work on Windows 7+, if you have 32-bit Windows, the original version should work out of the box. And if you have 64-bit Windows, some Russian guy made a proper 64-bit compile of it (that the original developer refused to accept for unknown reasons) that worked just fine when I used it on Windows 7 x64, and I got it running even on my 9841 VM back then (all the way to 9879). Might even work on 10036 but I'd need to try it.

Better so perhaps the efforts would be better put into making a generic library that can be used with emulators, one that supports all the various formats. So instead of every emulator having their own code for the floppy controller etc it could use an external library that also supported mounting of all common (and uncommon, pending on format documentation availability) formats. That way you don't need to focus on your own formats and just put the work into the emulator supporting the various other PC hardware... And it could then support your PEF format as well...

There *IS* libdisk that does just that. But the thing is, I still have to code support for the library into the FDC emulation and that would still take time. It also makes development less efficient, as if I find a bug with something inside libdisk, for example, I need to go to those maintaining that project and report the bug to them and hope they'll even care to fix it.

Sorry for all the rant, but I am sitting here planning for the new BetaArchive FTP tool I am coding (which will standardize a lot on the BA FTP) and I thought of standards in terms of floppy image formats...

Well, as I said, standards should be decided by the entire community and should balance between the need of preservation and difficulty of implementation.

[mrpijey]

IMG and FDI (assuming you mean Japanese FDI) are already supported, and I even made PCem-X support IMG better than other emulators as I've gone to great length to make images of XDF floppies load correctly. TD and KF are sadly still both poorly documented.

I can't do much about TD (I have some info about it, dunno if it's any useful), but I am working on getting some good specs about the STREAM format KF uses, as well as the CTRAW format. Both are however already supported by the CAPS library, which by itself should be fairly well documented for implementation.

While I understand your point, you have to understand that already the 0 and 1 FM or MFM bit stream is hard to implement, a magnetic flux format is even harder to implement, even more so when it's in an undocumented (or poorly documented at best) format. I also think storing the magnetic transitions is pointless. The data on the floppy is by definition digital. The magnetic transitions, in the NRZi format in case of a PC floppy, are just a way to store digital bits onto magnetic media in a way that it can then be reasonably reliably read back. Considering that what a normal floppy drive reads from the floppy are in the end the 0's and 1's of the FM or MFM stream, in the end that is the absolute minimum that has to be preserved. Anything else adds no real useful information.

Of course I understand it's hard, I am not questioning that. But remember, a floppy drive can read the magnetic flux on an "analog" level, which means (in theory, I have not seen this in practise) that some systems with dynamically programmable controllers (like Amiga, Atari etc) could read the flux levels manually and use that as a means of copy protection etc. Converting that analog data to digital could ruin the protection scheme. As I said, I am not fully aware of such systems used in commercial software, but during my time as a demo coder on the Amiga I've seen some crazy stuff some devs did with the drive controller, which included reading off the flux info off a floppy and use that as a means of protecting code. Of course, the PC doesn't have this kind of controller so it wouldn't matter in your case, but I am just explaining why it would be potentially best to allow support for a format that stores data on a much lower level.

And please note I never said I'll never implement KF or CTRAW formats - I will when there's enough documentation available. And when the rest of the emulator matures to the point of being able to guarantee the correct timings for floppy emulation (right now, it by far can't, the CPU code is still too slow, and the entire thing is single-threaded so the CPU code ends up slowing down everything else, the emulated FDC included).

Alright, fair enough.

Hence why we need to find a commonly agreed minimum that preserves every single bit from the floppy, while reasonably omits what is reasonably superfluous to make implementation easier. Yes, of course, having KF and CTRAW supported immediately would be perfect, but right now it's not the priority for me. One day in the future I am going to think about adding KF and CTRAW in, but right now, getting PEF to a reasonable degree of usability and implementing PCE's formats is as far as I can hope to get.

Well of course, it's your emulator and you're the coder, so it's your decision. I am just "requesting" features that would be beneficial from my point of view (being that of a preserver/software curator).

Well, hopefully I can implement PCE's formats and it can then lead to more people adopting them (which would in turn lead to even more emulators adopting them). But that's as far as I can hope to get in the near future. Anything further (KF and CTRAW) has to wait.

As long as you don't make the same mistake as others (by not properly documenting your format etc) I don't see why your format shouldn't have a chance, given that your format has an advantage over other more known and established formats. Remember, the KF guys for example knows their stuff, and even if their documentation is (for now) somewhat lacking the format itself is well thought of and future proof (i.e by storing as much info as possible, down to the flux level as highest possible resolution). Would be a waste to not utilize all that work into a good format. But as you said, you may support it in the future.

If your floppy emulator provides an interface through which user-mode programs can talk to it, then I can easily make PCem-X talk to it. Also, Virtual Floppy Drive does work on Windows 7+, if you have 32-bit Windows, the original version should work out of the box. And if you have 64-bit Windows, some Russian guy made a proper 64-bit compile of it (that the original developer refused to accept for unknown reasons) that worked just fine when I used it on Windows 7 x64, and I got it running even on my 9841 VM back then (all the way to 9879). Might even work on 10036 but I'd need to try it.

I've had some bad experiences with it on x64, and I wouldn't even consider running x86 edition of Windows anymore. And since the original developer has no plans of upgrading or further developing it someone else has to do it, if it's of any value that is. Of course the software would have a proper interface since it would be useless without it. But it's all in my head for now and I don't know if it even would be beneficial for any other purpose than pure learning.

There *IS* libdisk that does just that. But the thing is, I still have to code support for the library into the FDC emulation and that would still take time. It also makes development less efficient, as if I find a bug with something inside libdisk, for example, I need to go to those maintaining that project and report the bug to them and hope they'll even care to fix it.

Ah, there you go then. It would be a good idea to support it though, since it's open source you would also be able to get help by the community at fixing the bugs etc. And then you wouldn't need to reinvent the wheel yet again. But in the end it's up to you where you want to put your coding efforts into. We all just want the best emulator with the best features .

Well, as I said, standards should be decided by the entire community and should balance between the need of preservation and difficulty of implementation.

Well, that being said I bet the community want KF STREAM and TD support, so hop to it! (j/k). The difficulty of implementation is always the bugger though, and I understand if other features get priority. Would it be possible to modularize your emulator somewhat so you can get help implementing the features? Or going open source? I mean, with my experience in graphics acceleration and driver development I bet I could help you somehow, and that would also give you more time to developer the parts you feel needs priority, while others work on other parts...

Perhaps your emulator should have a plugin system like many other emulators have where you choose the various plugins for the different interfaces (video, floppy, I/O ports, audio, extensions etc). Then you can develop your core modules, and others could code theirs according to your plugin standards. Of course it would be tricky and take time to implement, but it would allow others to participate in development without actually having to have access to your emulator code. For example, one dev could code a plugin that properly emulated a plugged in GD5420 GPU, whereas an another dev could code a proper SB 2.0 implementation etc... that way you could "plug in hardware" into the emulator as if it were real expansion cards. Of course within the limitation of your CPU and bus limitations (what are your limitations, 386 instruction set and ISA bus?). That would make your emulator by far more dynamical in development and features and then you could focus on the core aspects of the emulator, i.e CPU and bus emulation etc.

Just an idea for the future .

[Battler]

PCem-X is open source. I post the source alongside the binary every time. I have to do it anyway, as per the requirements of GPL v2. And since the original PCem which my PCem-X is a fork of is itself GPL v2 licensed, I am bound to license mine under a compatible license, and I just decided to keep the same one.

That would make your emulator by far more dynamical in development and features and then you could focus on the core aspects of the emulator, i.e CPU and bus emulation etc.

Well, CPU and bus emulation I haven't dared touch yet. I still rely on Tom Walker's patches for that, since I don't by far know enough about that. Also remember, that originally I was simply submitting my own additions back to Tom. I went the fork route only well after there was enough friction between me and him that I felt a fork would have been better. I still want to contribute some of it back but I don't know what I could considering how little of it he'd accept.

(what are your limitations, 386 instruction set and ISA bus?).

Same as regular PCem, ie. all the way from the original IBM PC to an Intel 430VX with a Pentium MMX and PCI.

I've had some bad experiences with it on x64,

Then you didn't use the correct x64 version. The one that the original developer posted is an example of bad coding, and the Russian guy who made his own, better version, actually contacted the original developer about it and the original developer was all dismissive of it.

As long as you don't make the same mistake as others (by not properly documenting your format etc) I don't see why your format shouldn't have a chance, given that your format has an advantage over other more known and established formats.

You're mixing things up. I was talking about the PCE formats which are of, well, PCE, a different emulator by a different author. I am considering trying to plug that emulator's floppy code in and see if it can be got to work, however I don't guarantee anything at this point as I don't even know what license the other emulator uses.

[anormal]

hi guys,

my points @battler:

Did you read the drCoolZic documentation? and also the WIPs in softpres page?

I know it's a lot, and a lot of information there, it's not necessary for PC emulation, because as you already know, pc FDC is *simple* comparing with other platforms (amiga and
c64 specifically), but you'll get the point and general ideas about floppy preservation (not your interested, flux levels mandatory here), and disk img emulation (flux levels needed in some platforms, and *i think maybe* not necessary in PC).

I kindly recommend you read throughly this documents to have a broader view of the problems

I am not a expert myself (certainly softpres and other people, SCP developer, kryoflux people, drcoolzic) but read a lot of this and have some ideas

Disk image storing for later emulation is a very complex topic, much more than you could think at first sight: for example read last updates 3.70 in this atariSt emu
http://ataristeven.t15.org/Steem_370_coming_soon2.htm
or game analysis @ drcoolzic site

...

Levels of preservation (afaik this is not exact, some people could explain right):

flux level --> kf raw, scp raw, psi
analog->digital conversion, dpll decoding to bits ---> ipf??
mfm-fm bitcell level --> transcopy, ct raw?, pri?
only sectors and tracks with no extending disk information (no gaps, etc)--> td0, ana, cp2?, psi?
logical formats (fats, directory, boot sector) ---> raw, ima, imz,...
simple files compressed in a zip/rar

For what i read, you plan to make your format at the bitcell levels (as tc), right?

I am not sure (pc floppy protections must be studied very well) if for proper pc floppy emulation you need more than this, maybe you could run 99.99%, and for few software you need flux levels?, i don't know, and maybe in some months we could aswer to this with proper knowledge.

Now that i think... Flux information is needed to decode changes in recording density? errors in clock? crc errors in mfm format, etc... I am not sure if there are protections in PC using this. Probably yes.

As mrpijey said i am not going to tell you what to do, of course But as user of your emulator, will your format support anything more than other formats at your level don't support?.

If not, why develop another format? As mrpijey said, then you have users to convert to your format already existent formats? An online repositories store another format, and other emulators add code to support yours?

I have a lot of documentation about TC format, td0 format (it's not really complex, just some images are compressed and some no) and other formats, if you want i can compile all
of this (documents, printed webpages, notes from forums, source code) and send to you. Also i have (i think you already have also), many TC,td0,raw,ct raw, scp raw, images if you want to have them for testing.

As coder myself (tough i've not read pcem source and don't know internals) i know that if you have a good and complete structure of the floppy, the original data is not relevant as you only have to uncompress/decode the original data and put in the structure. Am i right?

If i were you, in this order:

- check FDC emulation precision, maybe you could check FDC emulator in Kryoflux source (cycle exact, most precise), not PC but nice to get ideas

- FYI: Amstrad CPC uses a nec 765 FDC, Sugarbox emulator http://sugarbox.free.fr/ supports raw reading, mfm conversion, etc... with the same FDC as PC, also CPC emu Caprice supports IPF

- implement your format/reuse PCE format for mainting data structures for mfm-bitcell emulation

- implement tc, ctraw -> to this structure loading

- minor formats as img/td0 could be easy to load to this structure

- when you are bored , implement raw flux loading with digital to analog conversion (pll implement,filtering, etc), drCoolzic has done an incredible work in this area with the tool Aufit, not for PC but principles are the same


Notes:
ima,imz,img are standard images, typical 360K floppy file of 368.640bytes
td0 files could be uncompressed, normal and advanced compressed (lzss), possibly copyqm and anadisk are the same format? all by Sydex

Interesting links:
- must read: http://simonowen.com/samdisk/cpc/ information about Amstrad CPC protections, also used a nec765 FDC so it must be equal to PC
Of course http://simonowen.com/samdisk/formats/ A LOT of formats are support, i think IS THE TOOL THAT SUPPORT MOST FORMATS OF ALL

- keirf disk utilities: https://github.com/keirf/Disk-Utilities/ (tools and source supporting: ipf,ctraw (kryoflux), imd (imagedisk), scp(supercardpro), img)

- libdsk (ima,td0) (uPD765a/i8272 emulation): http://www.seasip.demon.co.uk/Unix/LibDsk/

- drCoolzic document about AtariST protections (most complete doc): http://info-coach.fr/atari/documents/_m ... ection.pdf

- td0 support: first start in Dave Dunfield page: http://www.classiccmp.org/dunfield/img/index.htm (most complete and updated)
Also both http://www.willsworks.net/wteledsk.htm and http://erokhin.tripod.com/tdcvt.html contains algorithm and source for support all teledisk images

- supercard pro: format specs http://www.cbmstuff.com/downloads.htm, as you can see SCP (and KF) support multiple revolutions per track (needed for weak/fuzzy bits?)

- KF site with docs, source, etc

- latest Hampa's PCE emulator compile and source http://www.hampa.ch/pub/pce/pre/ - also PRI/PSI/PFI documents and sources for loading and decoding

---
Sorry the rant! but there is too much information and many decisions about all this stuff

[Battler]

Now that i think... Flux information is needed to decode changes in recording density? errors in clock? crc errors in mfm format, etc... I am not sure if there are protections in PC using this. Probably yes.

Well, changes in recording density maybe, but a PC FDC doesn't support multiple densities in the same track anyway. It does support different densities in different tracks by recording each track with a different data rate but it's the enough to have a byte for every track telling what format the track is in (FM/MFM, data rate). As for CRC errors, that again can be done at the bit cell level because the CRC is stored right next to the sector data. And clock bits are part of a MFM or FM bit stream (the whoel purpose of the bit stream format is to store the clock bits in addition to the data bits).

as you can see SCP (and KF) support multiple revolutions per track (needed for weak/fuzzy bits?)

Was that kind of protection ever even used in PC floppies? And nothing stops me from, for example, storing every bit stream twice, and in case of a bit differing between the two copies, it gets interpreted as a weak or fuzzy bit. But I'd need to know more about this protection first.

If not, why develop another format? As mrpijey said, then you have users to convert to your format already existent formats? An online repositories store another format, and other emulators add code to support yours?

I have a lot of documentation about TC format, td0 format (it's not really complex, just some images are compressed and some no) and other formats, if you want i can compile all
of this (documents, printed webpages, notes from forums, source code) and send to you. Also i have (i think you already have also), many TC,td0,raw,ct raw, scp raw, images if you want to have them for testing.

Well, I'd sure appreciate documentation about the TC, td0 and CT Raw formats and TC and CT Raw images I can use for testing.

[Darkstar]

I think if a new floppy format should ever succeed it would have to be a superset of all existing formats, i.e. it has to be able to represent all currently existing disk formats for all systems.

You cannot simply use the "lowest common denominator" and convert everything (e.g. existing sector dumps) into flux-level streams, because that would give a false sense of accuracy (is this really an authentic original dump? Or is it "just" converted from a sector-dump)

The basic unit of storage would probably be the track. This track could either be a simple set of "cooked" sectors (i.e. the user data), "raw" sectors (including sync marks etc.) or one of the types of raw trackdumps (bitcell, flux)

It should be possible to mix and match tracks with different "representations" or layouts in one image, to not overly bloat the image (what use are raw fluxdumps if your source image was only a sector-based dump?), so that only the tracks required for copy protection need to be stored in a more low-level representation. This also means that there must be a way to represent the various copy protection schemes based on unformatted tracks (i.e. random data) or deliberately destroyed tracks. Some systems use different write densities for different zones of the disc, that must be possible as well.

Track synchronization must also be preserved if possible (or at least if required by copy protection), so that the tracks are "aligned" to each other (this is only important for lux/bitcell dumps). This also includes support for hard-sectored disks etc.

Of course, the format should be expandable, backwards-compatible, and support metadata of any form and kind (text, binary, etc.). Bonus points if the format supports some sort of authentication, i.e. signatures directly in the image itself, to make it easier to assert the authenticity of an image.

Creating such a format would be an enormous task that probably requires many different people from many different emulator communities to cooperate, so that no edge-case will be forgotten.