Imagine you and your team have worked tirelessly to develop that video game you’ve always talked about. And today, it’s finally completed. Your final build completes and you finally have it: the most precious 10GB file of your life.
To your absolute delight, your game gains traction. After making the file available to download from your servers, you get tens of thousands of players downloading it every week!
There’s just one problem… everyone is reporting a bug. And not one of those “haha, look at this silliness” type of bugs. More like a “this is embarrassingly bad” type of bug. Not to worry! The fix is easy: just a few lines of code. You fix it ASAP and upload your new 10GB build with your fix. You feel relieved and sign off for the day, satisfied.
The next day you log in expecting to see a lot of “thank you"s and raving feedback in the forums. Instead, you are greeted with frustration. Users are annoyed that they installed the game already and now they have to re-download the entire file again. Not only that, there are complaints of slow or failing downloads due to the high volume of data that your servers cannot keep up with, as the entire player base eagerly tries to download the fix at once. How could a change to a few lines of code cause such a headache?!
Not surprisingly, there are better ways to handle this. Intuitively, for such a small update, you shouldn’t have to re-distribute the entire file again and pay all that network egress cost.
This class of problem has been the subject of research for decades. The solutions built in this field have applications in data backup and recovery, version control, and anything else that can benefit from data de-duplication and incremental updates of large data. I’m going to walk through the gist of this research and proven solutions in an extremely quick manner, as it pertains to the specific use case of patching/updating video games — just because that sounds the most fun.