Friday, February 26, 2010

Updater Design Decisions

My blog postings have been a little thin - but not because nothing is going on. Rather it's a combination of working on next-gen stuff that is still in heavy development, travel, and (perhaps due to the travel, perhaps due to lack of sleep, or perhaps due to certain forms of Brandy brought on site by a certain British developer who won't be named, but whose handle on the org starts with a P and rhymes with "Cropsman", cough, cough) a minor fever that I've just about recovered from.

A user asked me about the design of the X-Plane updater - so this post is only going to be of interest to the few authors out there who are creating installers for their add-ons and need to update. We try to keep the file formats for X-Plane simple, e.g. "this one folder is your airplane", so that if you don't make an installer, the user won't be overwhelmed with a complex unpack-and-install routine. But if you do need to install or update, well, here's what we were thinking when we developed our installer.

We had a few needs for our installer:
  • It had to be cross-platform to Mac, Windows and Linux.
  • It had to update existing products as well as install new ones.
  • The updates had to be minimal deltas, e.g. a demo can be 600 MB but an update should only be the 30 MB of files that really changed.
  • We wanted the updater to do delta updates from any old version. (Some update systems require installing a number of patches in sequence. We have users who buy the DVD and then update a year later; we wanted to let them update in one shot.)
  • Building the patches had to be really, really fast. We use our updater to publish our betas. So while X-Plane probably has 2-3 major patches in a single year and maybe 5 or 6 bug fix patches, we will (due to betas) cut perhaps 100 or more actual "updates".
  • On that last point, cutting separate updates by platform was a deal breaker, as it would triple the number of updates we had to cut. Having a demo install and an update be the same on the server was a big win.
Some of those needs are pretty specific to Laminar Research, but some might apply to a third party. There are commercial installers that will let you generate patches. If you don't generate a lot of patches, you can probably use existing tools - basically you'll spend a lot less time up front (since the tool set exists) but you might spend more time in the long run cutting separate patches by OS.

One of the first things we decided was to not use server-side technology or special protocol. That is, we don't need any kind of smart servers to run the updater - the updates are just files hosted over HTTP on standard rented or owned apache servers. All of the work is done in the client. We did this for a few reasons:
  • It lets us throw up the install anywhere - we don't have complex needs for what's on the server. Virtually any server will do. (As the company has grown, our server needs have grown too, so this is less of a concern now, but back then we had fewer servers in service, and were buying less overall server capacity.) If a server goes down, we can press another one into service about as quickly as we can move the file set to the server.
  • We're not server programmers. Coding the installer/updater on the client side let us leverage existing company technology, etc.
Here's what we came up with. The basic idea of the installer is very simple. Note that since the installer uses HTTP files, you can "watch" the installer simply by downloading the files that it downloads.
  1. First the installer goes to a master server and gets a "version list", which tells it what it's going to actually get and what (multiple) mirrors are available. This one master file is the only file that must be hosted at, and it allows us to change a very small file to press mirrors into service.
  2. Installs and updates are actually the same to us - it's a set of files that the user should have once the install or update finishes. This starts with a file directory that lists every file (by path) that needs to be downloaded, as well as its MD5 signature.
  3. The directory also contains the MD5 of every old version of any file in the version, and lists files that have to be deleted to update the version (e.g. the file panel.png used to have MD5 signature 2934b..23abc2 but is now removed).
  4. The client, once armed with the file directory, can now download all files in the directory (install) or compare the existing files on disk to the directory by MD5 and only download changes.
  5. Individual files are on the server in zip form for download and decompression.
That's pretty much it - you could write an installer in a scripting language using a tool like CURL or WGET - it wouldn't be very complex because the installer is really just a clever, painted download+unzip tool.

Dealing With Modified Files

There is a bit of complexity in this design that you might not need for a third party installer: handling modified files. Note that the directory contains the MD5 not only of the current version of the file (to detect when no update is needed and save bandwidth) but also to see when the user has modified a file that is "managed" by X-Plane.

Before the installer overwrites/deletes a file that you own, it compares the file's MD5 to the entire known list of MD5s for the entire version. If this file doesn't match a known old version, it puts up the warning dialog box that you have modified a file that is about to be overwritten.

While we recommend that add-ons use scenery packs and other "safe" ways to customize the sim, this helps detect when a user has gone in and edited the cloud textures in Photoshop, and prevents us from overwriting them without warning.


Steve said...

Interesting enough bit, Ben. And it gives me the opportunity to ask why the installer client is not smart enough to perform a rollback to a previous version. Why? This is especially useful if Austin's elbow bumps into a particularly valuable bit of code and breaks it while tweaking X-Plane to be "absolutely perfect." The rollback would be exactly the same as an update, deleting uniquely new files and re-replacing new files with their older versions. This would require an updater rewrite and time, to be sure, but as X-Plane gets ever more complex, it's continually getting more and more important to have this. What do you think....will we ever see this?

Benjamin Supnik said...

Hi Steve,

The installer can't roll-back because the history is not adequate - the history is the _union_ of all old versions, but each _individual_ old version is not fully tagged. Basically if the old version was fully hosted somewhere, the installer would be able to decode it.

Re: rollback and bugs...I think we're going to keep resisting this. Intentionally creating an environment where there is a soup of incompatible x-plane versions (e.g. 940 doesn't work like 930, so now we have half the user base on one, half on the other, and they don't work the same) doesn't solve the compatibility problems with add-ons ... it just makes the environment for authors even worse.

Anonymous said...

Can you give us some details about the next-gen stuff youre working on?

Maybe even a blog post?

Regards, MatthewS

Benjamin Supnik said...

Sorry, no. We're still in stealth mode.