BetaArchive Logo
Navigation Home Screenshots Image Uploader Server Info FTP Servers Wiki Forum RSS Feed Rules Please Donate
UP: 22d, 10h, 4m | CPU: 62% | MEM: 6003MB of 12227MB used
{The community for beta collectors}

Post new topic Reply to topic  [ 5 posts ] 
Author Message
 PostPost subject: [HOWTO] How PAR2 files work (& how to recover data using it)        Posted: Fri Apr 17, 2009 11:31 pm 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
7355
In relations to my data recovery process I'd like to take the chance to tell some of our members how parity files work. The parity files has been my salvation for verifying and restoring bad and/or missing files in the past and also during my last recovery session. Some of you may already know how it works but here it is anyway, a small section on what PAR2 files are, what they do and how they can help you save your files during a bad file copy, harddrive corruption or if you simply want to verify the integrity of your files.

--------

A set of PAR2 files are files containing parity information for the actual files you want to protect. Parity data is simply a piece of data meant to use with the main data to detect errors. If the main data has errors you can combine the parity data and good pieces in the main data to reconstruct and repair the file. It may be easier to illustrate this, so I've made some smaller pics:

Edit: Apparently the forum makes the images even smaller, barely unreadable now... I'll see if I can replace them with better ones soon...


Image
Green blocks are data blocks of a file, or individual files. Blue cells are parity data blocks.


This is an illustration of a full file or a set of files (same principle applies in any situation) including a par2 set. In my example you got myfile.iso as an example, but it can also be a set of files, like RAR files or any files. The parity files are extra files usually ending with a ".volX+Y.par2" where X is the par2 part number and Y is the amount of blocks the part recovers. So if you got a "myfile.vol3+4.par2" it means it's recovers from part 3 and is able to recover 4 blocks of data. The amount of total recovery blocks is determined when you create the par2 set. The more blocks you got the finer recovery options you will have, but also many more par2 files to keep track of. Use less files and you get larger par2 files and higher risk of losing all data if you lose a par2 files with many recovery blocks.

So, say your myfile.iso got corrupted when you copied it across the network. Or if you downloaded it using the bittorrent protocol (which uses data blocks from various seeders to construct the final file). and you didn't get all the blocks. Usually you need to delete the file since it's incomplete/damaged and thus unusable. My example also includes one of the parity files missing or damaged:

Image
The red cells are damaged data blocks or missing files.


But you got your parity set intact enough to be able to reconstruct the damaged files. What your parity software does now is that it uses all available parity blocks AND all available and undamaged data blocks/files to recalculate what parts are missing, and then reconstructs the file. Illustration:

Image


However this comes with some limitations! The parity files can only recover a limited amount of damage to the file/file collection. If you build a PAR2 set to cover 10% damage then it can only reconstruct a file/files as long as the missing data doesn't go beyond 10%. If your damage is more extensive then you will not be able to restore your data. (Don't mind counting the cells in the illustrations, it's mathematically unsound (damage is 5 cells, yet parity recovery is only 4), it's only meant as rough illustrations). You can of course choose how much parity data you want to build, but it will add to the overall size of the parity data. If you choose to build parity data that covers 20% of a 100MB file the parity data itself will take up 20MB (20% of 100MB). But these 20% will allow you to recover ANY damage of the set as long as the damage is 20% or less.. Remember that the individual damage of each file in a set doesn't matter, it's the entire PAR2 set that counts. If your parity data covers 250 files of various types the combined damage of all these files can't exceed 20%!

The parity files also includes the original filenames, so if you accidently renamed your files the parity software can automatically detect the file and rename it to the filename it originally had during the parity data creation.

PAR2 sets does not work with folders! Only individual files or a single file. The files can be ANY files, contents doesn't matter.

As a small foot note, a similar principle is used in RAID arrays. But instead of adding extra files it embeds the parity data in the file system and the parity is evenly spread out on all hard drives in the RAID array to maximize redundancy. Some archive formats (like RAR with redundancy options set also does it, spreading out parity data across the entire archive file (BetaArchive releases uses this option!) .

--------
Software to create and verify PAR/PAR2 files (my recommendations):

Windows:
  • QuickPAR: GUI driven tool to create PAR2 sets and recover data using them. Also supports the PAR1 standard. Supports CRC32 and MD5 verification files as well.
  • par2cmdline: command line driven tool for creation and verification of PAR2 sets. Great for using in scripts and batch files (see my examples below). Exists for a lot of platforms.

Mac OS X:
  • Split&Concat: Supports creation and verification of PAR2 files, also splits and joins files (great for usenet downloads).

--------
There are of course many other tools to make and verify PAR2 sets, but these are the ones I use on a daily basis. Feel free to add more apps if you find any good ones.

When I want to create many PAR2 sets across several folders I use a small batch file I created. It's crude but it does the work. What it does is it creates a PAR2 set in EVERY folder (first level only) found in the starter folder you are at. I.e if you run it in C:\Stuff it will create a PAR2 set in every first level subfolder, i.e C:\Stuff\Folder1, C:\Stuff\Folder2 etc.

makepar2f.bat
Code:
@for /D %%i in (*.*) do par2cmdline c -r15 -s384000 "%%i\_recovery.par2" "%%i\*.*"


-r15: Makes a PAR2 set covering 15% of parity data.
-s384000: Block size. Smaller blocks yields more PAR2 files but finer control of recovery options.
_recovery.par2: The filename pattern I use for all PAR2 sets. All parity files will begin with this.

There are many more options of course, experiment with them and see what works the best with your configuration.

--------
I am reserving the right to change this post if it needs updating. If something is too unclear let me know and I'll rewrite it to make it clearer.

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Tools: Alcohol120% (Portable)
Listings: BetaArchive Database (beta)
Channels: Discord :: Twitter


Top  Profile  WWW
 PostPost subject: Re: [HOWTO] How PAR2 files work (& how to recover data using it)        Posted: Mon Apr 20, 2009 8:09 pm 
Reply with quote
Donator
User avatar
Offline

Joined
Tue Aug 12, 2008 7:37 pm

Posts
2381

Location
United States
I did not even now what a par file was... anyways ill have to test this out and purposefully damage some files (sounds like fun :P)


Top  Profile
 PostPost subject: Re: [HOWTO] How PAR2 files work (& how to recover data using it)        Posted: Thu Apr 23, 2009 7:10 pm 
Reply with quote
Donator
Offline

Joined
Sat Oct 04, 2008 5:43 pm

Posts
1237

Location
Milky Way Galaxy

Favourite OS
Windows Server 2012 Dtc
When a BitTorrent file get corrupt, just right click, force re-check.

_________________
See my profile for my website link.


Top  Profile  WWW
 PostPost subject: Re: [HOWTO] How PAR2 files work (& how to recover data using it)        Posted: Thu Apr 23, 2009 9:53 pm 
Reply with quote
Administrator
User avatar
Offline

Joined
Tue Feb 12, 2008 5:28 pm

Posts
7355
motherboardlove wrote:
When a BitTorrent file get corrupt, just right click, force re-check.


I mentioned bittorrent because if you downloaded enough data but not all (perhaps a lack of seeds making it impossible to get 100%) you can use the downloaded par2 data to recover the rest. Traditionally you can just do as you said, but that requires seeders seeding all the parts you need, and it also requires you to download those parts. If you got enough par2 data you can recreate the missing data yourself, saving time and bandwidth.

Proper usenet releases always posts a par2 set just for this reason, in case any parts are missing or corrupted during download you can always recreate the missing data.

_________________
Image
Official guidelines: The Definitive Guide to BetaArchive :: Abandonware
Tools: Alcohol120% (Portable)
Listings: BetaArchive Database (beta)
Channels: Discord :: Twitter


Top  Profile  WWW
 PostPost subject: Re: [HOWTO] How PAR2 files work (& how to recover data using it)        Posted: Sun Jan 17, 2010 5:11 pm 
Reply with quote
FTP Access
Offline

Joined
Tue Feb 19, 2008 7:46 pm

Posts
292
PAR2 + SFV = Super combination, especially when your doing it on the good old' trusted news binaries.


Top  Profile
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 5 posts ] 




Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Search for:
Jump to:  

All views expressed in these forums are those of the author and do not necessarily represent the views of the BetaArchive site owner.

Powered by phpBB® Forum Software © phpBB Group

Copyright © 2006-2018

 

Sitemap | XML | RSS