Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

When I was younger, I literally thought PAR's were magic files. I had no idea how they worked, and from a distance it was magic.


"Any sufficiently advanced technology is indistinguishable from magic." -- Arthur C. Clarke

I thought the same thing when using PAR files. They're still useful today if you save things on media that can be damaged (CD, DVD, Blue-Ray) or across multiple multiple media.

Eventually, I decided to dig into the math behind it. It is a surprisingly simple principle:

Given polynomial of a degree X and an array of data points of size X, there is one and only one solution to the polynomial's coefficients such that it will pass through those data points.

So, stripe the data into bands of arrays, compute the polynomial, and compute additional data points of the curve, and save it with the original data. If you have at least the array's size of data points ( original array and/or parity values) and know the place in the list for each data point (thus which data is missing), there is one and only one solution to the polynomial equation. Once you solve the polynomial again, you can compute any point, including the missing ones. Again, because there is one and only one solution for the curve.

The devil is the math necessary solve the polynomials, which is why it is so computationally intensive.


> "Any sufficiently advanced technology is indistinguishable from magic." -- Arthur C. Clarke

"If anything seems like magic you're not asking enough questions." -- Mario Romero Vega


> They're still useful today if you save things on media that can be damaged (CD, DVD, Blue-Ray)

For writable disk media, there is also dvdisaster:

https://dvdisaster.jcea.es/


PAR files use ReedSolomon error correction which IMO might as well be magic.

Galois Fields are really awesome (and are related to CRC codes). The level of effort to learn is quite high. NASAs guide to ReedSolomon was amazing though

-----------

XOR codes are easier and sloppier. But are actually what's used today despite being imperfect.

Let's say you have A, B, C... Z as your data.

Parity#1 could be XOR(A, B, Z). If B were missing, Parity#1 XOR A XOR Z can back-calculate B.

Parity#2 can be a random set of all previous entries (including Parity#1). Etc. etc. etc.

Keep adding XOR parity codes until your probability of reconstruction is high enough to please you.

I believe this concept is called a Fountain Code.



Yes, this is it.

It's a very shallow introduction to Galois Fields but it's just barely enough to reach Reed Solomon encoding and understand error correction codes.

As I said earlier: it's a lot of math, even in this simplified form. Abstract conceptual math. But I do find this kind of abstractness very fun.


Also I’ve found that while most people (non-tech) don’t have a concept of XOR, they probably took basic algebra and understand 1+?=3

Arithmetic wouldn’t be a good implementation due to integer overflow (a problem XOR doesn’t have) but it’s helpful if you ever have to explain it to the less technical business person who you need to sign off on the purchasing decision.


that's how i used to explain what a nonce was to explain what all the computers were doing to "mine" bitcoin. and then explain "they're instead trying to get a number that has a certain number of zeros in a specific place"




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: