Clean Airport Layouts

I have blogged about correct airport layouts before, but let me bring the point up again, because it is so important:

• You must create structurally correct layouts (that is, vertices connecting to vertices, not lines) in order to get good rendering in X-Plane.  Just because the preview looks okay in WED doesn't mean your layout is correctly formed!

I wrote some documentation on the scenery site that describes the problems in more detail, with pictures.  

(I've been trying to create more permanent documentation - it is tempting to simply blog the issues because it's so easy to throw a blog post up, but after 110 blog posts in 2007, the scenery site is still very thin on the documentation front.)

Do Not Add Vertices To Make Smoother Curves

I really can't stress this enough: please do not go adding extra vertices in your layout to make bezier curves look smoother in X-Plane.  Why is this such a bad idea?  Let me count the ways!

1. X-Plane will fight you all the way!  X-Plane adds vertices to curves (to make them smoother) when it detects large errors.  If you have a lot of small curves, the errors are inherently smaller and X-Plane will add fewer points.  So the first vertices you add to your curve do almost nothing.  You have to add a huge number of vertices to get a marginal improvement in your curve.  In the meantime...
2. X-Plane will provide variable-quality curve rendering in the future!  Curve detail should be a user-controlled setting.  X-Plane has to run on a wide range of hardware; any time we can let the user pick rendering quality, this is a win, because it helps bridge the gap between the user who just bought a brand new Core 2 Extreme system with GeForce 9800 and the user trying to keep X-Plane 9 running on his G4 laptop which can't be upgraded.  When you add vertices, you take the decision about rendering quality out of the hands of the user, and force high quality on a user who may not be able to handle it. Adding vertices forces a decision of lower framerate on some users.
3. Adding vertices bloats the size of apt.dat.  This is not a huge factor for custom scenery, but is a factor for the default apt.dat.  Robin received a big pile of new airport layouts, and that's great.  But one risk is that the total size of user submitted data could get out of control.  For new layouts made with WED, vertices represent a big chunk of the data.  If you are increasing your vertices by a factor of 5x or 6x to improve tessolation, you are bloating the apt.dat file.
4. Manually adding vertices to smooth curves lowers the level of abstraction in the apt.dat file.  Any time we can have a high level abstract representation of scenery, X-Plane has the freedom to improve rendering in the future.  If your layout is made up of a large number of small curves (instead of a small number of large curves), X-Plane cannot tell that those small pieces make up some larger structure; in the future it may not be able to render those layouts as nicely as ones that are made with fewer control points.

In summary, please use the smallest number of vertices to create your layouts.  (But always add vertices to ensure that your T junctions are correct!)

I Hate It When My Padding's Not Integral

"Warning: padding is not integral" is a warning message that the airport sign code renderer can spit out when your airport is loaded.  Unlike all of the other warnings that you might see, this one doesn't actually tell you that there's a problem with your sign.  It's actually telling you that there's a problem with my code, and today I finally fixed it.  (The fix will ship in X-Plane 920, not X-Plane 902.)

The problem is that the official FAA sizes (0.76 meters, for example) suffer from tiny rounding errors when put through the sign creation process.  (Use 0.33 as a proxy for 1/3rd.  Divide something in thirds  and add it up.  You lose a tiny sliver, right?)  That warning message was the renderer noticing that it was being asked to fill in a really tiny sub-pixel area (which it then did, hence you can't see a problem from your airplane).

With 920, the sign heights are changed ever so subtly (by less than 0.1%!) so that they don't cause rounding errors.  This is the equivalent of calling the sign 0.33 meters instead of 1/3rd meter.

If this all seems like decimal point hell, well, welcome to my world!

Bottom line: you can ignore this warning, it'll go away in 920.

Irrational Sliders

I am reading Predictably Irrational, by Dan Ariely. It's a great read - definitely recommended - describing the consistent irrational biases that frequent human decision making.

The first chapter discusses our tendency to make relative, rather than absolute comparisons. When deciding whether a product is a good value, we will look at the pricing of similar models, rather than the actual relationship between the product and the money spent. (The implication being that a company can make a product seem cheap without changing its price by adding a second, more expensive but similar "decoy" product. Poof! The cheaper product is now a good deal.)

This behavioral tendency explains user reaction to the rendering settings, a subject that makes me irrational on a regular basis. :-)

Time to Change the Settings

The rendering settings will let you select a range of sim detail between some minimum and maximum value. These values are based on the software, not hardware - because we don't actually know how much load any given hardware can support (and with the interaction between settings, finding such a cap is basically impossible). We can only give you a range of choices and let you pick ones that work well.

When a new version of the sim comes out, we sometimes have to recalibrate the settings. If the minimum features the sim can support increase, the minimum setting will be mapped to a new, more expensive behavior. And if the maximum detail the sim can present has increased, the maximum setting will be similarly remapped. We don't have much choice - if we need more "range" on the slider we have to recalibrate it.

I Can't Max Them Out

Here's where human behavior comes in. Humans make decisions based on the relative comparison of easily compared things. Given properties that are harder to measure and easier to measure, we'll pick the easier one. Given a choice of a trip to Rome, a trip to Rome with free breakfast, and a trip to Paris, we'll pick Rome with the free breakfast, opting for the easy to measure relative value. (Is the difference between a trip to Paris and Rome really less than the value of a breakfast? Probably not, but it's a lot harder to evaluate.)

So when we recalibrate the settings, we inevitably here this complaint:

"I used to be able to set the sliders to the maximum setting and now I can't."

Previously I would have said "Why the hell do you care?!?!" -- if the new slider's 50% position looks the same as the old slider's 100% position, why not just set it to 50% and go home happy.

But of course that's not how we think - the immediately comparable is of immediate concern. Ironically we could make the sim less useful but more pleasing by limiting the maximum range of the sliders. Now more users could feel the joy of having everything "set on max" even if the ultimate utility of the sim is reduced.

This One Goes To 11

I'm not sure there's a way around this. The best suggestion I've heard so far is that if we could attach some kind of units to the settings, then at least there would be a quantitative indication that the user isn't losing some perceived value. But I suspect that even this misses the point; it doesn't matter that you're still getting 500 trees per square km - what matters is that you are getting the most you possibly can! (Perhaps this psychology also explains why people like to overclock.)

Austin tried to fight the psychology of "maximum sliders" by naming all of our settings absurd things. Ever wonder why "default" is the lowest object setting, and we almost immediately jump into "extreme", "too many", "insane", etc.? He was trying to fight a losing battle against relative expectations. The natural human behavior is to pick some relative position for calibration, and based on that, every user who has to put objects below the center setting is going to be unhappy about having to use "lower than average" settings. Austin's naming convention may be silly, but it does actually do a little bit to fight this.

Food for thought: how does having multiple levels of reflections change user expectations?

Authors: Always Check Log.txt

If you are developing an add-on for X-Plane, you should always check the log file (Log.txt) after running.  (Remember, the sim must quit before the log file is completely written to disk.)

There are three times you should check the log:

1. Any time your content doesn't look the way you expect.  (Perhaps the error is described in more detail in the log file.
2. Any time there is an error (especially the "a problem with the package X" message, which is always accompanied by details in the log).
3. Right before posting your work as a last check.

Posting errors to the log gives us a way to provide verbose feedback to authors when the sim hits a problem without making the user experience too horrible; minor errors are logged and major errors are mentioned to the user only once per package and logged.  

The flip side of this is that if you are working on content, you need to seek out these error log messages; the alternative is to have the sim quit every time something goes wrong.

Limits On Texture Paging

I seem to be in a philosophical mood these days with my blog posts...thought for the day: the human mind easily goes from the specific to the general. Our brains are generalizing machines, pattern matchers finding the rule in the noise.

My preference in creating new scenery-system features is to make them very limited, and my reasoning is: our brains don't go backward very well.  We do not go from the general to the specific.

Now you might think: when making a scenery-system addition, the best thing would be to have a general feature, more useful because it can be used everywhere.  But I say: the most important thing is to fully understand the feature - otherwise the feature comes out buggy. 

(Consider the piles and piles of bugs and weird behaviors that you get when combining OBJ animation with OBJ hard surfaces.)

Since the human brain doesn't go from general to specific well, it is hard to start with a rule ("let's allow feature X in all parts of the scenery system") and comprehensively derive all of the implications; it is human nature to be surprised later by some unintended side-effects.

It is always easier to extend a feature later to its natural full implications than to declare certain uses illegal later, after authors of planned or started trying to use the feature in that way.  If the generalization of the feature makes sense, extending it is often quite painless.

Texture Paging - Scope For Now

Texture paging is the ability for X-Plane to raise and lower the resolution of scenery textures dynamically as you fly.  This means more VRAM used for nearby things and less for far away things.  In practical terms, this reduces VRAM used by orthophotos by down-sampling the far-away textures, making larger orthophoto scenery packages possible.  As you fly, the sim reloads some textures at higher resolutions and some at lower.  The cost of the features is the load time while you fly, which burns up some extra CPU cores.

It is my hope that we will productize some very simple texture paging in the next major patch of X-Plane 9 (that would be 920, not 902).  But the usage will be pretty specific:

• Texture paging will only be available for .ter and .pol textures (we can extend to other scenery types later if it makes sense).
• Texture paging will require changing the .ter and .pol files (X-Plane will not automatically analyze your scenery to see what can be paged.)
• Texture paging will not be available for ENV scenery.
• If you share textures and texture page, the results will probably be really bad and cause chaos.  Be sure to use only one .ter or .pol file (and reference that text file only once in the your DSF definitions section) if you want sane paging.  We can extend paging to shared textures in the future, but for now orthophotos are the intended target.

I am also deferring work on dataref-driven textures; we'll get there eventually, and the infrastructure from the pager will make it easier.  But dataref-driven textures really need to be available in a lot more places - it's a bigger, more complex feature* and I can't keep adding scope to 920.

Make New Meshes!

While paging will be available for both overlays (using .pol files) and base meshes (using .ter files) I strongly, strongly recommend going the base-mesh .ter route.  RealScenery sent me their new "State of Washington" package to use as test material; I was pleasantly surprised at the high framerate.  Part of that comes from them using base meshes and not overlays. 

Overlays cause the sim to draw the scenery twice (first the old scenery, then your overlay), burning a lot of pixel shader and fill power.  Base meshes simply replace the old mesh which is at least twice as efficient.

(I'm just going to keep beating the dead horse of base meshes because I believe that the sooner everyone moves toward base meshes, the more bang for our hardware buck everyone gets.)

* In particular, remember that texture paging happens on threads.  But datarefs can come from plugins that are not threaded!  Insert anarchy here...

More Scenery Features

Since X-Plane 9 went final I've been going in about 5 different directions with the code, and as a result my posts have diverged pretty far from my charter within the company, namely scenery. Here's some of what I'm looking at on the scenery front:

Texture Paging

Texture paging is a new feature where X-Plane can change the resolution of orthophotos while you fly. The big limitation of orthophoto-scenery right now is that X-Plane loads every orthophoto at the maximum user-defined resolution when the underlying DSF/ENV is loaded. This can cause X-Plane to run out of virtual address space and crash. With texture paging, only the nearby textures are loaded at high resolution.

Far away textures may be low res, but you'll never notice because they are far away (and thus you're seeing them over only a few pixels on screen anyway.

The cost of this technique is that textures are being loaded over and over and over. But this cost is made less painful by two features:

• DDS texture loads are very fast and cheap, especially at very low resolutions.*
• With the next major patch, texture loading will be entirely on a second core (if you have one) and can even span multiple cores.
  

This feature will require modification of scenery packs in that you'll have to add some "paging" information to your .ter files; I will try to get a preliminary spec posted soon. Because you only have to modify text files, it should be possible to create "paging patches" to existing DSF orthophoto sceneries that are fairly small.

I do not know if paging will be available for .pol files. My guess is yes, but I reiterate that using .pol files for large areas of orthophotos is a bad idea! Make a new mesh!

Improved Multi-Core Performance

This is going to be an on-going process through the entire v9 run and beyond, but basically with every change to our rendering engine (and we make some serious modifications with almost every patch to keep it modern and competitive) we try to move toward using the new hardware (meaning machines with 2-4 cores) more efficiently. Some of this will be in 920, although my guess is we'll need at least one more patch after 920 to really see the improvement.

Tools

It's going to be a little bit before I can put more time into the various scenery tools. My top priority is to keep up with user feedback and work on MeshTool. Hopefully we'll also declare WED final soon. But for now, since I am working on cockpit and airplane modeling features, my next set of work will probably be for the airplane authors.

Shaders

I do want to put additional shader work into v9. I realize I am at the edge of provoking a bunch of rants in the comments section, so let me save you some time:

"Hey Ben, stop working on eye candy and create some more tools. I don't want a shader feature that makes my 1.2 ghz G4 with a GeForce 2 MX run any slower. You should finish the tools so they do exactly what I want!"

Okay, I admit, that was totally unfair...there is a lot of truth in the complaints about shaders vs. tools.

• I really do try to keep an eye on system requirements, particularly once we shipped. I'm going to try to prioritize shader features that improve existing rendering techniques, rather than introduce new rendering techniques, so we don't seriously lower fps during the version run. But also bear in mind that shaders can be turned off, and there are users who have GeForce 9s and such.
• Tools are very important, hence the effort to get MeshTool out. But tools without engine work aren't very useful either; most of the engine work we do is needed to keep performance up so that new scenery (made with those new tools) don't bury the sim. For example, MeshTool without texture paging would be a dead-end...you could easily make a MeshTool scenery that you can't fly.

So in planning what goes out when, I look for clumps of features and tools that can go together to make some sense: WED to use apt.dat 850, texture paging to go with MeshTool. It wouldn't make sense to defer texture paging to make the next tool while MeshTool is waiting for engine enhancements.

* A DDS already has all of the smaller-version pre-compressed textures in the file. So loading a DDS at low res involves loading a small amount of data from disk to memory and sending it to the graphics card.

By comparison, a PNG file only contains the maximum size, so to load a PNG at low res, we load the largest size, shrink it on the fly, then apply DDS compression.

How Much Can I Chew?

Austin posted another State-of-the-Union yesterday, and he was good about only mentioning things that are fairly close to completion (e.g. the panel texture optimizations) as well as things that are general (e.g. threading optimizations). After a long beta, when 9.0 went final I sort of went crazy and started a whole series of new projects at the same time; here are some things that I have in progress:

• Major rewrites to the texture management code for both better orthophoto handling and better threaded performance.
• Moving the sim to unicode and true-type fonts, with a new system for translating the application's strings. (I'll post more on this soon - thanks to those who have already volunteered to work on translations!)
• Working on new shader technology to take better advantage of DirectX-10-class hardware.
• A big pile of airplane features to complement what was added to 9.0.

That's a bit much for the next patch, so it's likely that only some of these things will actually make it into the next patch, and I'm not sure what you'll see. A lot of sim work goes in as a series of small independent pieces; only the last parts of a feature are user-visible.

For example, the first part of texture work was simply rearranging how the code was structured to make new things possible. Change to the user experience: none. The second part changed threaded handling of textures, which at least shows up as performance improvements. But both the stage for new features later.

So even if a lot of the above doesn't make it into the next major patch, a lot of ground work is going in, setting us up for features later.

Multi-Core Texture Loading

In a previous post I discussed the basic ideas behind using multiple threads in an application to get better performance out of a multi-core machine.

Now before I begin, I need to disclaim some things, because I get very nervous posting anything involving hardware. This blog is me running my mouth, not buying advice; if you are the kind of person who would be grumpy if you bought a $3000 PC and found that it wouldn't let you do X with X-Plane (where X includes run at a certain rendering setting, framerate, or make your laundry) my advice is very simple: don't spend $3000. So...

• I do not advocate buying the biggest fastest system you can get; you pay a huge premium to be at the top of the hardware curve, particular for game-oriented technologies like fast-clock CPUs and high-end GPUs.
• I do not advocate buying the Mac Pro with your own money; it's too expensive. I have one because my work pays for it.
• 8 cores are not necessary to enjoy X-Plane. See above about paying a lot of money for that last bit of performance.

Okay...now that I have enough crud posted to be able to say "I told you so"...

My goal in reworking the threading system inside X-Plane for 920 (or whatever the next major patch is called) is, among other things, to get X-Plane's work to span across as many cores as you have, rather than across as many tasks are going on. (See my previous post for more on this.)

Today I got just one bit of the code doing this: the texture loader. The texture loader' job is to load textures from the hard drive to the video card (using the CPU, via main memory) while you fly. In X-Plane 901 it will use up to one core to do this, that core also being shared with building forests and airports.

With the new code, it will load as many textures at a time as it can, using as many cores as you have. I tested this on RealScenery's Seatle-Tacoma custom scenery package - the package is an ENV with about 1.5 GB of custom PNGs, covering about half of the ENV tile with non-repeating orthophotos.

On my Mac Pro, 901 will switch to KSEA from LOWI in about one minute - the vast majority of the time is spent loading about 500 PNG files. The CPU monitor shows one core maxed out. With the new code, the load takes fourteen seconds, with all eight cores maxed out.

(This also means that the time from when the scenery shifts to when the new scenery has its textures loaded would be about fourteen seconds, rather than a minute, which means very fast flight is unlikely to get to the new area before the textures are loaded and see a big sea of gray.)

Things to note:

• Even if we don't expect everyone to have eight cores, knowing that the code can run on a lot of cores proves the design - the more the code can "spread out" over a lot of cores, the more likely the sim will use all hardware available.
• Even if you only have two or four cores, there's a win here.
• Texture load time is only a factor for certain types of scenery; we'll need to keep doing this type of work in a number of cases.

This change is the first case where X-Plane will actually spread out to eight cores for a noticeable performance gain. Of course the long-term trend will be more efficient use of multi-core hardware in more cases.

Why Is the v9 Global Scenery Split In Half?

The global scenery comes in two packages in version 9: -global terrain- and -global overlays-. The global terrain package contains the base meshes (with beaches); the global overlays contain roads, forests and objects.

Why is this scenery split in half? The answer is unfortunately not "so you can replace the base mesh but keep the overlay 3-d stuff." That would have been clever, but I must admit I didn't think of it at the time; MeshTool didn't exist and people just weren't making base meshes.

My actual goal was to make it cheaper to replace a significant number of overlays. I don't know if we'll ever do this, but one of the obstacles to patching global scenery is the file size; we can only hope to replace a fraction of the files during a version run before the web update size gets too large. But most of that size is in the base mesh. With the base mesh and overlay split, we could potentially replace more overlays.

(Note: we did not actually issue any DSF replacements during the v8 run, and I don't know if we will or will not during the v9 run. The only thing I am sure of is that if we provide replacement v9 DSF tiles, they'll be a free download, like all v9 patches...if you buy v9, you get everything.)

The fundamental problem with replacing the base mesh but not the overlays is that the scenery system provides no good way to do this. The Global Scenery folder is always scanned after the Custom Scenery folder* so you'd have to install custom scenery into the global scenery folder with the right file name to get access to the overlay content.

I'm not sure what to do about this yet; the trend in scenery development is for authors to want more control to replace individual parts of the system; the overlay system provided part of that.

* Users with v9 beta DVDs will have the two global scenery folders in the Custom Scenery folder. But -- the sim detects this and simply treats them as if they were in the Global Scenery folder, ignoring alphabetic ordering.

Don't Put Non-ASCII In Your Scenery Files!

I don't know how much of a problem this is yet, or how much of a mess it's going to make of people's scenery. Here's the background:

• ASCII defines 128 character values, mostly letters like A-Z. With ASCII, you can write English and that's just about it.
• The byte that ASCII is stored in on all modern computers can store 256 values.
• Clever people got the idea to put some more letters in the other 128 values to create characters like é and å.
• People defined different "codepages" that have different sets of charcters in those "upper 128" slots. So one code page might be good for French, another for Russian.
• Modern software uses unicode characters, which have a lot more than 256 values, and can thus hold all sorts of characters in one string.

Code pages were around for a while, but they're not a good idea. The problem with code pages is that the same numeric values are used for different letters. The result is that a correctly written Russian document, when converted to a different code page, looks like gibberish. And if you want one document with both French and Russian, well, one code page doesn't do you much good.

Now X-Plane's handling of non-ASCII characters is pretty poor in version 9.00 (and all previous versions). It will draw ASCII and take keyboard input from ASCII but not much else. If you hit the é key on your foreign keyboard, probably nothing will work.

But it turns out there is one way to use foreign characters in X-Plane - I just discovered it tonight. If you use Windows and your system's codepage* is set for a foreign language, you can use those foreign language characters in an OBJ file to name a file on disk with the same name. In other words, you can have textures named été.png and it will work.

Sort of. If you then change your system to work in Russian (which changes the code page) your texture will stop working. The reason things stop working is that the file system uses unicode; that is, the OS knows that été requires a Latin character set that's French friendly, but X-Plane is using Russian since the system's set that way. The result is that the file system has no way name the file in Russian and we fail to load the texture.

So using the "high 128" characters from your system's code page to make non-ASCII characters is a bad idea because your scenery won't work on other people's computers.

But it's going to get worse in the future. X-Plane is going to start using UTF8 in a lot of places. UTF8 encodes unicode into one byte characters by using more than one byte for non-ASCII characters, but as a result it uses the "high 128" character codes for very different things. été.png in UTF8 comes out quite different.

I'm not sure how we'll handle this yet (use UTF8 in the scenery system or have some kind of backward compatibility). But for now I can only advise one things: use ASCII only for your file names. In fact, a good guideline for filenames for the scenery system is to use only numbers, letters, and the underscore.

The Relationship Problem

I just finished about 15 pages of emails, mostly to Andrew McGregor (who is the very first MeshTool user) and also Benedikt Stratmann (whose x737 is on the bleeding edge of plugin-based aircraft) and AlpilotX (we all know about his forests). Probably all three are wondering how the hell I have time to write so much on weekends. (The answer is of course that my frisbee game got rained out. Foo!)

In the meantime, probably about 300 other people who have emailed me in the last few
months are wondering why the hell they have heard nothing from me. My in-box looks like a mail server exploded. It's not pretty.

So let me blog for a moment about the "relationship problem". Simply put, there are two of us (Austin and myself) and about a thousand of you (third party developers doing cool and interesting things with X-Plane) plus significantly more users, some of whom have some very weird tech support problems.

In this environment, our algorithm for who gets "developer attention" is pretty broken and subject to total thrash...there is a huge element of random luck (who emails me when I am recompiling the sim vs. debugging a nasty bug).

I'm aware of both how hard the task Austin and I face and how frustrating it is for a third party developer because I've been on both sides. Before I worked for LR, I was a third party and I was always astounded that Austin couldn't remember what we talked about last week.

Then I started working for the company and saw what it's like. Imagine sitting at a train station watching the trains go by* (at full speed, not stopping) and someone says "last week I waved to you out the window and you waved back, remember me?"

So I would advise three things to the neglected third party:

1. Be firm - you may need to ping us again because at busy times we can't always keep track of who has asked for what.
2. Be patient - if you need something the week we're burning DVD masters for a second time (because the first set failed at the factory) then you're going to have to wait.
3. Don't take it personally...a lack of a response usually indicates overload inside the company, not a poor opinion of your work!

This blog post has rambled enough, but it may feed well into the next one.

* This analogy is totally stolen from "How Doctors Think" by Jerome Groopman - he uses it to describe the task of primary care physicians trying to spot the early signs of a very rare illness among a fast-moving train of patients who are almost entirely healthy. I strongly recommend this book particularly for Americans - we need to understand the forces at work in shaping the quality of our medical care!

Road Overlays and Draping

X-Plane 9 allows you to put roads in an overlay DSF, but this isn't actually that useful yet.  The problem is that roads are specified using MSL elevations - that is, they are positioned in absolute space.  So you have to know where the ground is to build a road, which defeats the whole idea of an overlay.

This dates back to X-Plane 8.0 - originally all scenery load was done when a DSF was loaded, with the sim paused.  In that condition, keeping scenery load fast was a high priority.  So I set roads up to be "pre-draped" (that is, their heights are pre-determined when building the DSF). Back in the day it was a win.

Now we have multiple cores and DSF loads in the background, so we very easily could "drape" a DSF...but it gets weirder.  A road's sub-type is specified in the DSF - and sub-types are used to define bridges.

Now that's not incompatible with a "draped" scheme - imagine a road whose height is specified AGL, so a height of 0 means "run the road along the base mesh" and a height of 10 means "10 meters above the base mesh."  Now you would set the road type to overpass as the height went from 0 up to 10 meters, then back to 0, forming a bridge.

The only problem is what happens if the ground is really weird looking.  In practice what we want to do is keep the road's slope as constant as possible and build out bridges based on that. The effect can be hard to see, but if you look at bridges in the US scenery, you'll see that sometimes the banks to a river slope down but the road stays horizontal, forms a bridge, and then stays flat as the banks come up again.  This happens because our road processing code (in the DSF builder) tries to limit road slope.

That's not an easy effect to create in an overlay...you'd basically have to increase the road AGL at the rate the base mesh's AGL is decreasing (if only you could know that).

So the questions I am debating now are:

• Are AGL-draped roads useful if we don't know what the base mesh will be like?
• Does it make sense to have some other abstraction to help smooth roads when the base mesh is bumpy (and can we make something that is both simple and useful)?

Like most philosophical things I debate, this isn't scheduled to go into the sim that soon. Improved shaders, a more complete airplane SDK, and a better texture pager are all ahead of road improvements, which pushes roads out quite a bit I think.

The Global Scenery Folder

Let me answer two questions right now:

1. If you bought version 9 beta DVDs, you will not have to buy new ones when version 9 goes final.
2. If you bought version 9 beta DVDs, you do not need to manually move around any of the global scenery files that our installers put on your hard drive.

Now about this global scenery folder...

X-Plane stores its scenery in packages - a scenery package is a folder that contains all of the files needed for a given scenery element, whether custom or default. In X-Plane 7, scenery packages were only used for custom scenery, and always lived in the Custom Scenery folder directly next to the X-Plane application. This provided an easy way for users to drag third party packages into a single location to install them. (Before the custom scenery folder, installing scenery could involve a lot of "put this file here, and that file there".)

X-Plane 8 added a second location, the Default Scenery folder, which is hidden away inside the resources folder; it acts as a repository for scenery packs we ship with the sim, such as the one that contains various airport elements, and the default road packs.

In X-Plane 7 the global scenery lived directly in the resources folder in its own home; in X-Plane 8 the global scenery lives in packages inside the default scenery folder...this allows us to use one piece of code (the package loader) to load all scenery, custom and global.

The only real differences between the default scenery and custom scenery folders was priority and intended use; custom scenery is loaded with higher priority than default scenery (so you see your custom add-ons replace our default work), and the intention is that the default scenery is for Laminar to update via the web-updater and custom scenery is for you to put things into that you download off the web, etc.

With X-Plane 9 we are introducing a third location for scenery packages: the "Global Scenery" folder. It will live next to the Custom Scenery folder and X-plane application and be the location for global scenery. Its priority is intermediate - higher than default but lower than custom.

Now here's the funky thing: the X-Plane 9 beta DVDs place the global scenery into the custom scenery folder. If you've ever looked at the ranking of scenery packages, you'll see that they come out lower priority than all custom scenery, regardless of the alphabetical sorting that is usually used. The sim recognizes the peculiar placement of the global scenery and effectively labels it "global".

Future DVD pressings will simply place the global scenery in the new global scenery folder. Thus we get to the original two answers:

1. You won't have to buy new DVDs if you have the betas; the sim will continue to work with either the beta or the newer layout for global scenery for the entire v9 run.
2. Because both layouts work you don't need to move anything around. I suggest you not move any files to lower the risk of breaking things.

Scenery Tools Release

I have posted the first "combined" scenery tools release. Basically ObjView, ObjConverter, and DSF2Text (now named DSFTool) are packaged together with DDSTool (for making DDS images from PNGs) and the new MeshTool (for making base meshes); a UI application called XGrinder provides drag & drop support for most of the converters.

The tools releases will be by month (e.g. March 2008) and each release will contain the latest for all of the tools in the release. WED and the AC3D plugin will continue to be separate downloads.

I have also posted another snapshot of the source code, and generally I'll try to do a source code post whenever I do a tools post to keep the two in sync.

The AC3D plugin has been updated; I hope to have a new WED posted in the next few days.

X-Plane 9: What Comes Next

X-Plane 9.00 is going to be going final pretty soon - we're on RCs right now. But this is hardly the end of the road; rather it's the first step for a number of new development areas. What comes next?

Tools: I am working on tools releases now; a few users already have alpha copies of MeshTool. My hope is to have some new tools posted tomorrow, with more by the end of the week

More rendering engine/shader work: X-Plane 9 only begins some of the shader work we want to do; more engine enhancements will be coming in 910. Like 9.00, more advanced rendering will consume more hardware, and will be scalable via rendering settings.

Systems and Airplane SDK: X-Plane 9 introduced a lot of new features for airplane modeling, but there's still more to do. I hope to have the panel region system and advanced lighting options all fleshed out for 9.10. Austin is doing a lot of work on systems modeling. We'll be making more new airplane-related datarefs to tie it all together.

Orthophoto Scenery: I don't know if this is going to happen or not, but we get more and more requests for large-orthophoto-scenery support in X-Plane; MeshTool is only going to increase that desire by making it possible to cover large areas with orthophotos without sacrificing framerate. So I would like to do some work on the texture pager, but I do not know if I'll be able to do that in time for 9.1.

My hope is to get 9.1 into beta early so that we can start getting third party aircraft authors involved with trying new features. If you are working on an advanced airplane, keep in touch!

Version 9 - Probably Too Late

If you have a third party add-on and you haven't tested it against X-Plane 9, well...honestly it's probably a little bit too late to fix compatibility bugs now. We're in RC, and only changes to fix critical bugs are going into the sim; everything else will have to wait for 9.1.

So if you have an add-on and you haven't tested it against 9.0, for crying out loud, do it now! Submit the bug anyway - if we can't fix it for 9, we can fix it for 9.1. But...after almost 14 weeks in beta, we've got to close this build up.

BTW a minor side note on fuel pumps...

In X-Plane 864, an airplane will run even if the electric fuel pump is off. (However, if the fuel pump failure is triggered and the electric fuel pump is off, either by switch or electrical-system failure, then you're out of fuel.)

X-Plane 900RC1 restores this behavior; in beta 25, an electrical failure would turn off all sources of fuel. This is wrong for a plane like the C172 where gravity can feed the engine.

In the long term X-Plane does need a more complex abstraction (e.g. does this plane have gravity feed or engine drawn fuel, or engine driven pumps, etc. etc.) but for now the 864 model, while inaccurate and incomplete, causes less problems than beta 25, which was simply wrong for a number of planes.

The Limits of Orthophotos and Meshes in X-Plane

I get asked a lot about the limits of meshes and orthophotos in X-Plane. I'll try to answer this, but the answer isn't as simple as most people expect.

Texture Limits and Orthophotos

The maximum single texture size in X-Plane 8 is 1024x1024, and in X-Plane 9 it is 2048x2048.

I believe the maximum number of unique custom orthophotos that can be attached to a single DSF is at least 32768.

In practice, that number is pretty useless because X-Plane loads all textures for a DSF at the highest user-allowed res when the DSF is loaded. That means you tend to load a lot of textures. Every system is different and drivers have a lot to do with RAM efficiency, but generally you'll run out of virtual address space and crash the sim before you can attach 32768x2048x2048 of pixels.

X-Plane has no limits on how the texturing is applied - that is, you can use your 2028x2048 texture to cover an entire tile or a single meter. So again, the limiting factor on the resolution of your orthophotos is how much total area you want to cover and how much RAM you can spend (remember RAM is also used for mesh complexity, 3-d models, etc.).

You do not need to have enough VRAM to hold all loaded orthophotos; the video driver will paeg the textures into VRAM. Virtual address space is the limiting factor. How far you push it depends on a lot of subjective things:

• If you expect your users to also run with a lot of trees, 3-d objects, cars on roads, and some plugins, you can't use a lot of RAM.
• If you expect your users to have /3GB in their boot.ini and use nothing but your add-on, you can use a lot more RAM.

Generally the size of the DDS texture on disk is a good proxy for the virtual memory that is required to hold your textures.

It should be noted that these limits on texturing (due to X-Plane blindly loading a lot of stuff at once) affect all scenery types: objects, draped polygons, very complex airplanes, plugins, and not just terrain mesh orthophotos.

Getting Past the Texture Limit

It will take a future extension to the rendering engine to get past the current limits. Basically X-Plane will have to load textures at lower resolutions when they're farther away. I don't know when that is coming, but when it happens, it will increase the total amount of image data a DSF mesh can contain, because the limiting factor will be how much data is in the small area the user is looking at (since the rest can be stored at much lower res for far-away views). At that point the limiting bottleneck will be resolution (smaller means more data at once), not total image data.

Mesh Limits

Unfortunately, limits to the mesh are even more vague than limits to texture usage. X-Plane uses an adaptive mesh - basically you can put your vertices wherever you want. So the highest resolution you can achieve might be much smaller than 1 meter resolution, but you can only do this for a small area before the total mesh size gets too big. But this is okay - the intention of DSF is to let you put a lot of detail where you need it.

I believe that once again memory provides the first limitation to the mesh. That is - you'll run out of memory loading your insanely huge mesh long before you hit a limit to the DSF container structure. And once again, even the RAM limit isn't a hard limit because that virtual address space is shared with texures. Your mesh density limits actually go down when your textures go up because it's a zero-sum game.

Estimating Memory

Here are some ideas on how to estimate your memory footprint:

• Run X-Plane over ocean to get an idea of the baseline memory use that the sim needs without extra scenery.
• Load your mesh without textures (move the textures away) to find the cost of the mesh itself. (I am going on the assumption here that you can rescale your mesh using whatever mesh generation tool you're using).
• The size of DDS textures is a good proxy for the memory used.
  

MeshTool: The Seeds Are Sown

Last night I created the "seed" files for MeshTool. Let me explain what these files are and why we'll need them.

MeshTool is a wrapper around our irregular-mesh generation code. It will allow third parties to create base mesh scenery without having to create triangulations. Just like DSFTool saves people the work of having to encode DSFs (with point pools, command lists, and all that ugly stuff), MeshTool saves people the work of having to create their own triangulations.

MeshTool is a low level tool - you provide a text input file and some data. It's designed to be an engine underneath tools like PhotoSceneryX, not an end in itself.

MeshTool will create "default" land-use terrain that approximately matches the global scenery, water, and custom orthophoto-based terrain. You (or a program you use) provide a text file that describes the boundaries between custom photos, land, water, and airports. You must also provide a SRTM-style HGT file for elevation.

How does X-Plane know what land-use should go on what terrain? That's where the seed files come in. Our global scenery is generated from a set of rules that take into account morphology (land height and slope), approximate climate, and general land use. You provide the terrain shape via the HGT file, and we provide you with a seed file that contains climate and land use for that DSF tile.

Why do we provide the seed file rather than letting you find and create climate data? Well, our rules are tuned for a very specific pair of data sets; by providing the exact climate and land use data that we use, we assure that the rules files work correctly. The purpose of MeshTool is not to customize land use terrain, and we do not provide a mechanism for it. The purpsoe of MeshTool is to let you put orthophotos and new coastlines into the base mesh.

The good news is: seed files are tiny. They are typically 4 kb-8kb each; the entire data set is 322 MB total. That's because the climate data is only 3x3 per DSF and the land use is only 121x121.

I hope to get MeshTool into some kind of testing within the next few weeks; if you are a programmer and would like to feed MeshTool from your own program, please contact me and perhaps I can arrange an alpha copy. I will also post the seeds as soon as I can.

No Answer Does Not Mean Go Ahead

Every now and then I see a comment in an X-Plane forum somewhere to the extent of:

"Joe Author made this great scenery pack for FS2K4, I tried to contact him about a port. I got no response, and the pack is free anyway, so I've posted my conversion."

Simply put, you can't do that, at least not in the United States. Copyright law is very clear on this subject: if you don't hear back from the author, the default is that you do not have permission to create a derived work.

(The fact that the original package was "free", meaning cost zero dollars, is not at all relevant. The author retains his rights to his own work even if he doesn't charge money.)

A simple thought experiment reveals why it has to be this way: if I was giving away my new program as a promotional period and went on vacation, and you decided to post a derived work because (1) it was free and (2) you hadn't heard from me, I would have no way to stop an illegal use of my work that I did not ever want (nor ever indicated that I wanted). "Free" plus "no one is home" is simply not a high enough bar to protect authors.

Algo-Gen

I may be fighting a pointless and unwinnable linguistic battle, but I have to try. People very often refer to the default city buildings in X-Plane as "auto-gen" but by any reasonable definition of "auto-gen" they are not really auto-gen at all.

Now these are all made up computer terms, so we can't really check the dictionary. But "autogen" scenery (short for automatically generated) usually refers to scenery that is created by a flight simulator itself, usually while you fly, and usually by placing 3-d detail in places that match the base terrain. This exists in FSX, and existed in X-Plane up to version 7.63.

X-Plane 8 doesn't have autogen!!!!!!! X-Plane 8 has scenery that is generated by computer programs, but X-Plane is not the computer program that is doing it. When you see a ton of buildings piled up in New York City, that is not becaues X-Plane looked at the New York city base terrain and said "hrm - some buildings would be nice."

What actually happens is we analyze New York City when we create the global scenery (before we ever burn the DVD masters) and the DSF generator places all of those buildings in New York City. X-Plane simply gets a huge list of buildings from the DSF and draws them.

I am going to try to coin the term "algogen" (algorithmically generated) to describe these buildings that (like autogen) come from a computer generating semi-random buildings from input data, but unlike autogen, algogen is a process that runs once before the scenery is made.

So how is algogen and autogen different?

• You can't change the pattern of algogen building placement by editing files in the sim. The algorithm has already been run! You can replace the buildings using an overlay (that excludes the base) or by using a library of models to substitute models.
• We are trading data size for computation. The DSF is bigger because it lists the location of every building in New York, even if they were just algogen buildings, but the job of placing those buildings is less difficult because X-Plane does not have to check each building against each road. That has been done in advance.
• Changing the scenery via an overlay doesn't change the algogen! Add an airport via an add-on and you have to exclude the buildings. (But if you send that airport to Robin, the next global render will include it and the algogen will skip the airport automatically.)

Note one of the interesting results of algo-gen: X-Plane can't tell the difference between an alg-gen building and a hand-placed one! They're all just objects in a DSF. The fact that algo-gen buildings disappear with lower settings is because the sim/require_object property in the DSF header tells the sim which objects are important, and our generator always signals the buildings based on obstacle data as important. But algogen as a process is not visible to X-Plane!

And that's why I'm spending so many words on trying to distinguish between "algogen" and "autogen" - because the processes are fundamentally different, they're very different for scenery authors to work with. As a result, authors coming from X-Plane 7 or FSX will be very surprised if they try to understand X-Plane in terms of autogen....they won't be able to find the autogen config files, and the autogen buildings won't react to other scenery changes, because they're not actually autogen at all!

Algogen is a classic pattern of "precompute" vs. "compute-while-fly". Generally precomputing gives authors more flexibility (in our case, we have an obj engine that can handle a lot of objects, so authors can make their own objects of the same density as algo-gen with the objects placed anywhere) at the expense of making it more complex to edit the existing scenery (edit the mesh and the algogen doesn't change).

When we started the v8 scenery, two things pushed me toward precomputation:

• In the past, changes in X-Plane's rendering engine had broken third party add-ons. So a precomputation strategy (by getting the scenery code out of the sim) means that the sim is doing less "interpretation" and thus the interpretation of scenery is less likely to change.
• We wanted to focus on performance, which means getting computation out of the sim whenever we could.

Now that last point isn't quite as important as it used to be...when we were doing the design (during mid X-Plane 7), dual core for everyone wasn't on the radar, so the penalty for complex computation while flying is lower (and thus we have more expensive in-flight computation, like forests and completely draped bezier curve-based polygonal pavement).

But I think precomputation is still useful. Even with dual core, the algorithm that places X-Plane's algo-gen bulidings can take one to two minutes for a 1x1 DSF tile on a very fast computer. That's still a load time that's out of the question for us; even on the second core, the DSF wouldn't be "ready" in time for you to fly it. So one use of precomputation is to run algorithms that are more expensive than you can have in real-time. (That algorithm to pack objects inside an irregularly shaped polygon made by roads and land features is not fast.)

More importantly, precomputing does give us a nice advantage in the use of storage data. We ship about 50-60 GB of final scenery, but the source data is well over 100 GB. When we run the algogen algorithm, we have access to the full set of source data: coastlines, elevation, and land use before any simplification is done and any data is thrown out. So we have the potential not only to do a more complex analysis, but to do the analysis on a larger data set.

The down-side of precomputing is that if integration of all data is saved until sim time, there is the potential for third parties to contribute separate data to the sim via add-ons and still have the integration of those data sets work well. This doesn't always work out - see complaints in online magazine reviews about combining orthophotos and new road grids in FS2K4...they don't integrate because neither of those types of resources can be integrated to match the other in real time. But autogen still does a much better job than algogen at this; algogen basically has to be recut when other data changes. (And that is our intention - if you change the road grid, exclude and replace the objects too!)