Thursday, July 02, 2009

The Will to Rewrite

FSBreak interviewed Austin last's an interesting listen and they cover a lot of ground. A few comments on Laminar's approach to developing software:

That Code Stinks!

Austin is absolutely correct that we (LR) write better software because neither of us are shy about telling the other when a piece of code stinks. But I think Austin deserves the credit for creating this environment. An "ego-free" zone where people can criticize each other honestly and freely is a rare and valuable thing, in many domains, not just music.

When I first came to LR, Austin created this environment by responding positively to feedback, no matter how, um, honest. When I first came to LR 100% of the code was written by him and 0% by me. Thus if I was going to say "this piece of code really needs to be different", it was going to be Austin to either run with it or try to defend his previous work.

To his credit, Austin ran with it, 100% of the time. I can't think of a single time that he didn't come down on the side of "let's make X-Plane better". That set the tone for the environment we have now: one that is data driven, regardless of who the original author is.

I would say this to any programmer who faces a harsh critique of code: good programmers write bad code! I have rewritten the culling code (the code that decides whether an OBJ really needs to be drawn*) perhaps four times now in the last five years. Each time I rewrote the code, it was a big improvement. But that doesn't make the original code a mistake - the previous iterations were still improvements in their day. Programming is an iterative process. It is possible to write code that is both good and valuable to the company and going to need to be torn out a year later.

A Rewrite Is Not A Compatibility Break

Austin also points to the constant rewriting of X-Plane as a source of performance. This is true too - Austin has a zero tolerance policy toward old crufty code. If we know the code has gotten ugly and we know what we would do to make the code clean, we do that, immediately, without delay. Why would you ever put off fixing old code?

Having worked like this for a while, I am now amazed at the extent to which other organizations (including ones I have worked in) are willing to put off cleaning up code organization problems.

Simply put, software companies make money by changing code, and the cost is how long the changes take. If code is organized to make changing it slower, this fundamentally affects the financial viability of the company! (And the longer the code is left messy, the more difficult it will be to clean up later.)

But I must also point out one critical detail: rewriting the code doesn't mean breaking compatibility! Consider the OBJ engine, which has been rewritten more times than I care to remember. It still loads OBJ2 files from X-Plane 620.

When we rework a section of the sim, we make sure that the structure of the code exactly matches what we are working on now (rather than what we were working on two years ago) so that new development fits into the existing code well. But it is not necessary to drop pieces of the code to achieve that. I would describe this "refactoring" as straightening a curvy highway so you can drive faster. If the highway went from LA to San Francisco, it still will - just in less time.

In fact, I think the issue of compatibility in X-Plane's flight model has a lot more to do with whether the goal is to emulate reality or past results. This debate is orthoganal to refactoring code on a regular basis.

* Since OBJs are expensive to draw and there are a huge number of OBJs in a scenery tile, the decision about whether to draw an OBJ is really important to performance. Make bad decisions, you hurt fps. Spend too long debating what to draw, you also hurt fps!


Concerned X-Plane Dude said...

As a former devdog myself, Ben, I understand the desire for neat, tidy code. And adding efficiencies is good too. It just seems that too many little things creep in to the "new" X-Plane code that simply should not break. New decimal increments of the simulator should break new ground and repair nagging old bugs, not add new ones to existing functionality. Under the low overhead manpower situation that Laminar is in, sir, you and Austin don't have the luxury of a huge staff to validate your efforts. I only recently decided to participate in the Beta process. I was delighted that Austin chose to add a feature that I lobbied for. Bully good! But so much frankly silly stuff has emerged in the 9.3 Betas I have to wonder if you guys weren't exercising the will to rewrite just a bit over much. And that, sir, makes it Alpha testing, not Beta testing. Austin's recent radio interview depicts a man that would be very uncomfortable with a disciplined development process, where one picks a target, achieves it, perfects it, and moves on. Thanks to 9.3, X-Plane is loaded with new idiosynchrasies, like cu clouds that rotate with the view camera, and as of this writing, a simple panel view movement command that is no longer functional. How do, sorry, stupid things like this creep into the code? Austin bragging that he never tests his Windows executable before releasing it to the public says it all. His confidence is perhaps well placed in his compiler, but what of other confidences he has, and perhaps you have, that are misplaced and result in needless frustration? Right now Laminar needs to be on it's best development behavior as sim enthusiasts that want to stay with the latest and greatest migrate from FSX. Do you get the impression that's where you guys are at? Just a quick glance at the posts I'm seeing in the online forums suggests otherwise. Are your internal checks and balances what they should be?

Anonymous said...

X-Plane is an amazing piece of software. I'm a career software engineer, so I have a solid understanding of what goes into an application like this. It's very impressive that X-Plane has been developed by such a small team -- obviously the developers are very talented folks.

However, the release and QA process is extremely poor for a commercial consumer product, and it does hurt your users (even those who don't participate in the betas). I understand the desire to avoid having a lot of overhead and process -- those things slow development down and generally tend to hamper excellence. But process isn't binary. It isn't as if you either have heavy, bureaucratic process or none at all. It's a continuum. And even a small amount of process would be a huge improvement in X-Plane.

For example, how about a rule stating that no new features go in once you've reached beta? The point of beta testing is to work out bugs in new functionality, not to keep adding features and trying out new things. We're now at Release Candidate stage and you're STILL adding functionality (e.g. "A handful of new gps commands to control the software Garmin-430"). That's a serious misuse of the term "release candidate," IMHO. I'd also suggest a rule that you actually try all of the major functionality of the app before releasing a new beta (or especially a new RC). The breakage of cockpit scrolling between RC1 and RC2 is inexcusable. I don't see how that could have gone out the door if anyone had actually used that version for a significant period of time before sending it out.

I don't think these represent onerous process. They're just about excercising due care to actually provide a solid product to your customers. I have heard the objection that people who want a solid product shouldn't be playing with betas or RCs. That's true, but it doesn't discount the value of the changes I suggest. First, if beta testers are playing with new functionality or struggling with major breakage between betas, they are likely not fully testing the rest of the app (e.g. anyone who uses the 2D cockpit is pretty much unable to use and test RC2). Second, the amount of churn in the supposedly final versions of the app prolongs the time until final release and also increases the likelihood of bad bugs in the final release.

I'd also like to suggest that the willingness to break things is generally good, but it could be done in ways that are less disruptive to users. For example, it seems like every release lately has completely changed some major things about the way aircraft work (e.g. the recent back and forth over how the engines work), making huge numbers of third-party aircraft (and sometimes even the included aircraft) unusable. This is totally unneccessary. First, it's inexcusable to keep going back and forth publicly (in the form of releases) over how such a major part of the software works. If there's a problem, figure out the solution, test it, make sure it's the right choice, and THEN release it. Second, even very lightweight versioning information in the aircraft files would make things a lot easier on users. Obviously from a user's perspective, the best outcome is for X-Plane to fall back to the old functionality if an old aircraft is loaded. However, I understand the desire to avoid keeping all that cruft around in your code. So how about at least giving users a warning like, "this aircraft was based on an old version of Plane Maker. The engine response will not be correct until the aircraft is updated." And give aircraft developers an easy way to upgrade.

The bottom line is, you make your living from your users. It's just a matter of respect to them to try to provide a solid, quality product. That might mean adopting a few development practices you don't like, but you owe it to the people who pay you.

I want to reiterate -- I think you guys are doing great work. I do not mean to play that down. But I think it's time for the process to mature a bit. This isn't an open source software project, so it's time to stop acting like one.

Anonymous said...

Oh geeze...! I'm one of the harshest critics, one of the most disappointed users at some of the arbitrarily stupid decisions (bomb smoke, key flight controls...) and I don't think it's a big deal. If the beta process is the way this lean organization is going to move forward then so be it. If a user decides to participate in the beta process then he'd better do one of two things:

1) Have a workable backup, or...
2) Delay upgrading until the first user reports are in.

Though the betas can jerk you around a bit, overall I'm pleased with the progress and process, brushing off the problematic ones and enjoying the great intermediate builds I've come across.

All I'd consider asking for is a better explanation of what's changed in the release and how to use the new features in the wiki, where an example or an extra sentence or two could save me a few hours of grief.

I'm currently enjoying rc1 and bypassing rc2, but I'm gratified some attention is being paid to the arrow keys, which are somewhat bizarre in 3D cockpit mode. I expect something good will come out of this, or we'll just have to get used to the changes. Or Laminar will have to put up with incessant complaints.

Anonymous said...

honestly I've always feared the worst for you guys since Aces fired 99% of their employees.

game engine design is serious business and there is a LOT of stuff you need to know. I envy that you know all of Austin's secrets!

Keep up the good work.. and needless to say even microsoft has resorted to including nothing more than "windows.h". Just remember to test the network once in awhile.. or at least once win7 is released! I doubt your old compiler will curse at you for using such things as "string.h"..