Plausible, Realistic, Procedural and Algorithmic

Alpilotx pointed me toward a thread on the org discussing Austin's work on the weather system. The thread turned into a bit of a he-said-she-said with regards to Outerra and whether it could some day be combined with X-Plane.

This blog post will be a discussion of various general approaches to scenery and the trade-offs we have to consider, e.g. plausibility and realism, procedural vs. algorithmic and data driven design. But first, a brief note on Outerra. As I have said before, we are already aware of Outerra, so there is no need to email us. The bottom line is that we have a set of mostly done features for X-Plane 10, our goal is to finish X-Plane 10, and we are not even spending one brain cell considering putting a new rendering engine into X-Plane while we are trying to get 10.0 done.

Defining Some Terms

One of the problems with comparing scenery system approaches is that a real productized approach to scenery rarely fits into a perfect bucket or matches a single theoretical techniques. So here are some approximate terms, designed to generally describe an approach. They're not going to be perfect fits, and even the definitions will fluctuate in different contexts and forums.

• We can say scenery is plausible when it looks like it might exist somewhere in the world. Plausible means that roads don't go straight up over a cliff, trees don't grow in the ocean, etc. In other words, plausible scenery is scenery where absurd things don't happen. Plausible scenery is great when you don't know what an area should look like. A lack of plausibility is often a bug.
• We can say scenery is realistic when it correlates closely with what is really present at a given location on the Earth. So if there really is a lake behind my house, realistic scenery has that lake. Plausible scenery might have a lake, a forest, or something else believable for where I live (the Northeastern United States). A giant sandy desert would not be plausible for my location.
• We can say scenery is procedural if the detail in the scenery comes from some kind of algorithm that produces results. For example, a fractal coastline is procedural.
• We can say scenery is data driven when the detail comes from some source of external input data. Our mountains are currently data driven - that is, the mountain shape basically comes directly from the DEMs we use.
• We can say scenery is artist driven if the look of the scenery comes from art assets created by an art team.
• We can say scenery is algorithm driven if part of its look comes from the transformational process that converts data from one form to another.

(I'm sort of drawing a line in the sand here with procedural vs. algorithmic, but what I'm trying to contrast is a program that generates 'information' out of thin air vs. a program that creates information out of other information. For example, in X-Plane 9, European capillary roads were procedural. We had no real data, so I wrote an algorithm that made them up in a manner that was consistent with underlying terrain. In version 10, these roads will be algorithmic; we take OSM data and then do some processing to make it suitable for X-plane. This is definitely a line in the sand kind of definition.)

So Are We Plausible or Realistic?

So the first question is: is the goal of X-Plane global scenery plausibility or realism? The answer is: a bit of both. Austin's posts on the subject virtually always bring up plausibility. The reason for this is simple: he is not too worried about the amount of realism we've put into the scenery, but he is not happy with the bugs. He wants the bugs gone. So every time he and I speak, he says "and make sure it's plausible!"

But we're not going to remove realism just to fix plausibility bugs. I expect that the next global scenery render will be at least as realistic as the last - that is, we're going to use better data and we're not going to make up data where we had real information before.

There are limits to realism. We don't expect the global scenery to ever be as realistic as a custom scenery package for a small matter. But realism does matter. Part of the joy of flying in a flight simulator is seeing the real world. Where we can have more realistic global scenery, we consider it to be a win, and we are always looking to be more realistic than the last render.

Plausibility for the version 10 render is going to take two forms:

1. Bug fixes. Any time something screwy happens, it's not plausible. Sometimes these are code bugs that must be fixed, and sometimes they are data conflicts. For example, the water data sasys "water" but the elevation data says "hill". Combine them and you get water going up a hill. We have to write code to resolve this, somehow.
2. We are reworking the way cities are rendered, because even at their best, the old approach, procedural buildings with algorithmic roads over land class photos, did not look plausible, even at its very highest setting. So this is a feature request to fix a plausibility problem.

Algorithmic or Procedural

I've discussed this before (and forgotten about the post). But to expand the discussion, we need to consider not only algorithmic and procedural data processing, but whether we are driven by procedural generation, input data, assets created by artists, or some combination. (In practice, all systems require a mix of data, art assets, and procedures and algorithms, it's a question of the blend.)

I've been working on global scenery for a few years now, and over time I've come to appreciate the importance of artist input (via art assets) into any scenery process. Simply put, if you want scenery to look good, you need to make it reasonably straight forward for people who are good at making pretty pictures to control the look of your visual results. A few years ago I viewed the scenery process as strictly a question of data conversion and visualization, but now I see it as finding a way to merge art assets and data into a cogent final product, with the art assets being used in a way that the artists can control. In practice, this often means making sure that the art assets come in a format that artists are comfortable with or can learn without too much pain.

As I said in the previous post, our approach is becoming more algorithmic and less procedural as higher quality source data becomes available. (For example, we don't have to generate European roads when we can import and reprocess them.) But our approach over time has always been heavily artist driven. By this I mean: our input data is algorithmically processed into a final form that makes sense only in the context of art assets, and we have a pretty good idea of what those art assets will look like when we design the algorithms. To use roads as an example again, our task with OSM is to convert OSM road data into a road network that will visualize nicely with road art assets created by an artist.

Procedural Compression

One way to view procedural scenery is "creating lots of information from little or no information". But another way to think of it is as a compression technology. As was correctly pointed out on the org forums, you use less storage specifying the overall location of a forest than you do specifying every tree individually. The compressed form (store the forest location) can be equally plausible. It will be less realistic if the original tree locations were based on real world data, but it will be equally (unrealistic) if the original tree locations were procedurally generated. Put another way, pushing procedural processes out of the scenery generation process and into the flight simulator makes DSFs smaller.

When I first started working on X-Plane 8 DSF scenery, not only was DVD size a factor, but so was load time; we had one core and it wasn't a very fast core. Anything we could do to make loading faster, we did. Thus we pushed a lot of work into the scenery generation process, including procedural processes, to keep load time down.

Times have changed; we now have dual core machines as a baseline, and often quite a few more cores. Thus over time we are starting to move procedural processes back into the simulator, trading load time (which runs on multiple cores) for generation time and file size. So perhaps a more accurate statement would be: our scenery generation process is becoming more algorithmic and less procedural, and X-Plane itself is becoming more procedural. This is driven both by more input data (which must be processed up front) and more compute power on the host (which lets us shrink file size, and thus use DVD space for other things).

X-Plane 10

Here's how this plays out in practice in version 10:

• Some (but not all) of the building placement work* has been moved into X-Plane; a bit of expensive precomputation is still done at DSF generation time.
• Some (but not all) of road processing has been moved into X-Plane; a lot is still done at DSF generation time.
• Where possible, we are moving from a multi-layered approach to terrain to a pixel-shader-based approach to terrain. This cuts down overdraw and uses the GPU more efficiently. (The simplest example: in X-Plane 8 and 9, cliffs have separate terrains from hills. In X-Plane 10, a single terrain sits on both the cliff and the hill and changes its appearance based on the actual slope; this texture change is computed by the GPU.)

In other words, X-Plane 10 is making the logical evolution to better balance the computing resources we have to improve plausibility and realism.

Curved Roads

At this point I can say with 99% confidence that X-Plane 10 will feature bezier curved roads. In X-Plane 9, a road is a line segment; you can simulate curved roads by using a lot of line segments, but the global scenery roads are pretty chunky.

X-Plane 10 allows for a road to be a bezier curve, allowing the specification of smooth curves with a small amount of data. This sets us up to trade off visual quality and performance using a rendering setting.

A few notes for authors:

• Like all of the new v10 road features (and pretty much all of the new v10 scenery features), you don't have to use bezier curves in your roads. They are there as an option if you want them.
• X-Plane 10 will not make curves for you; road data that is defined as line segments in the DSF will be rendered as line segments. (This follows the principle that DSFs contain pre-processed scenery data, and the sim shows DSFs exactly as they are written.)

Pay No Attention to the Documentation

The DSF specification alludes to bezier curved roads; this "old way" of encoding curves was never supported in the sim - all versions of X-Plane ignore this data. The "old way" was how we thought we might do curves some day.

The version 10 curve encoding is different; the "old way" will continue to be ignored in version 10. So: do not use the DSF spec to try to make curved roads now. I will post detailed documentation on curved roads once version 10 is available to authors.

A Cliff Shader

I have been stingy with pictures of next-gen global scenery for one reason: it's really hard to get a nice shot of the global scenery that doesn't show unfinished features. With something like global lighting I can zoom in and show just the new trick, but with global scenery, I can't take a picture of a new house without showing a city block that looks funky due to a bug and a road that isn't finished. Posting a working shot of the global scenery where some sub-systems have bugs and artifacts would just freak everyone out.

I figure if it's obvious that the shot isn't a production shot, I can get away with posting it though.

A lot of the times when I work on the rendering engine, it is with test textures like this. Our art team does their best to hide the seams between different art assets, so that the scenery looks like one continuous world. The problem for me is that the better they do, the harder it is for me to tell if the underlying shaders are doing what they should do.

So alpilotx sent this test: it's all of the Innsbruck area painted with a test texture. What's new and interesting here is that the flat, hill, and cliff areas are all shaded by a single shader that selects between multiple textures (and rotates the textures) based on the underlying mesh.

We are adding the cliff shader to version 10 for a few reasons:

• Often we can get better cliff and hill definition by processing in the shader than by painting different triangles with different textures; our ability to control the transitions using different .ter files is limited.
• Using one slope-sensitive shader saves over-draw and triangle count, which makes the DSFs faster and smaller.
• Some day we may have the GPU distorting mountains on the fly to make them more mountainous. If we do, we need the GPU to also apply the correct textures; if the cliff areas are precomputed then they won't respond to GPU distortion.

A Global Scenery Squawk List

This post is a bit of an experiment...we'll see how it goes. When we cut the global scenery, we do a number of validations: we manually inspect some of the DSF tiles, we examine the Earth orbit textures (which are derived from the DSFs) as a quick way to look roughly at all of the tiles, we have a number of internal consistency checks in the generator, and we can compare our output to some of the input data (e.g. did we lose all of our water) to look for gross bugs. But we don't have enough resources to manually examine all 18,000+ tiles in detail inside X-Plane.

So: if you have found a defect in a global scenery tile in version 9, please list the tile in the comments section of this post. I will try to give these "previously broken" tiles priority in manual inspection.

Note that the bugs we can expect in version 10 will be very different than in version 9 because we've really changed the global scenery generation process in nearly every important way. The underlying algorithms are heavily rewritten and data sources are very different. The goal here is to simply find areas that might have a higher probability of weirdness.

Let me try to be clear about what constitutes a broken tile and what does not. Please read this definition six or seven times before you post a comment.

• Do not report bugs of inaccurate data. If your favorite road is missing, or the coastline is in the wrong place, don't report that here. That is not a defect in the rendering process; rather it is a limitation in the source data. I am not trying to collect a list of data inadequacies. We have already selected the new data for the new render. "Stuff is in the wrong place" is not a bug for this list!
• Really weird patterns are of interest. For example: I have seen some reports of very long thin bars of land going perfectly north along the edge of the tile, or a comb of mountain and valley, again perfectly vertical. These are bugs in the production process and I do want to know about them!
• Do not report errors in the placement of the 3-d layers (trees, roads, etc.). Since the 3-d layer is being completely rebuilt for version 10, none of these defects will be present in version 10. (They'll be replaced with a totally new set of weird behaviors!)
• If an airport is too bumpy to use (with "flatten " turned off in the sim), report this only if the terrain around the airport is grass. Basically the grass means that we tried to "fix" the airport in the v9 render and failed. If there is no grass (the airport is over forest or city) and it is bumpy, do not report it - that means it was added to apt.dat later.
• Do not report mismatches between the airport shape and the grassy patch; this is just the apt.dat file being updated.

If you have something to report (basically incorrect flattening of grassy airport areas and really weird coastline/terrain bugs that are clearly programming errors) please include three pieces of information:

1. The DSF tile, e.g. +42-072
2. The nearest ICAO of an airport near the bug (e.g. KBED)
3. A very short description of what went wrong (e.g. "tall thin vertical lakes running through the entire DSF").

Please only post defects of the above nature in the comments section of this blog post; to keep things clean I am going to nuke any off-topic comments on this post.

EDIT: see the first Squawk, Arista's report on LOWS. This is a perfect report: it includes the DSF tile, the ICAOs, a brief but clear description, and it is a bug in how we process the data, not the data itself.

Procedural Or Algorithmic

A quick thought on procedural vs. algorithmic scenery: there was some discussion on the X-Plane dev list about procedural terrain; the Outerra screen shots have stirred up interest.

There is a fundamental difference between what you see with Outerra and what we do with our global scenery:

• Typically procedural scenery is based on the idea of 'amplifying data' - that is, making a little bit of data look more interesting by applying a recursive algorithm to "generate" detail.
• Algorithmic scenery involves taking a large amount of unique input data and translating it; the detail comes from not losing information in the source data.

The key difference is whether the resulting scenery comes from a process that "creates" information through a 'procedure' or "translates" information.

Our scenery process does a bit of both, but we are more in the algorithmic camp than procedural camp; the global scenery is produced from hundreds of GB of input data, and we are constantly looking for better input data to create more interesting and accurate output scenery.

In fact, I would say that we are becoming more algorithmic and less procedural. In version 9, the urban roads in non-US cities are "procedural" - an algorithm generates them from the terrain data, an algorithm, and some random noise. For version 10, we are importing OSM.

One thing I have noticed in the work on version 10 global scenery is that we've finally crossed a line. With version 9, the question was 'what is the best data we can get'. With version 9, the question is 'how much information can we keep'; we've reached a point where the resolution of our input data is so much higher than what can go on DVD, that it's a question of filtering down, not synthesizing up the resolution.

OSM Tilings

This is two pictures of "tilings" of OpenStreetMap for use in global scenery. I downloaded a OSM new planet extract about a month ago; in the 11 months since I last grabbed it, the data size has grown 56%! The new, larger file required some changes to my extracting code. After much debugging, I was able to see this in QGIS:


The first picture is 1x1 tiles, which are derived from the second picture (10x10s). You'll see some "ragged" edges. This is because the cutting scheme leaves whole roads of interest in one piece even outside the tiling bounds. Later, more sophisticated code crops the road when the actual DSF is built.

The OSM processing tools are part of the open source scenery tools; I will get my changes checked in to source control over the next few days, although my code is only one of dozens of programs for bulk processing OSM.

No Raster Land Use Data

The X-Plane version 8/9 default scenery uses raster land use data (that is, a low-res grid that categorizes the overall usage of a square area of land) as part of its input in generating the global scenery. When you use MeshTool, this raster data comes in the .xes file that you must download. So...why can't you change it?

The short answer is: you could change it, but the results would be so unsatisfying that it's probably not worth adding the feature.

The global scenery is using GLCC land use data - it's a 1 km data set with about 100 types of land class based on the OGE2 spec.

Now here's the thing: the data sucks.

That's a little harsh, and I am sure the researchers tried hard to create the data set. But using the data set directly in a flight simulator is immensely problematic:

1. With 1 km spatial resolution (and some alignment error) the data is not particularly precise in where it puts features.
2. The categorizations are inaccurate. The data is derived from thermal imagery, and it is easily fooled by mixed-use land. For example, mixing suburban houses into trees will result in a new forest categorization, because of the heat from the houses.
3. The data can produce crazy results: cities on top of mountains, water running up steep slopes, etc.

That's where Sergio and I come in. During the development of the v8 and v9 global scenery, Sergio created a rule set and I created processing algorithms - combined together, this system picks a terrain type from several factors: climate, land use, but also slope, elevation, etc.

To give a trivial example, the placement of rock cliffs is based on the steepness of terrain, and overrides land use. So if we have a city on an 80 degree incline, our rule set says "you can't have a city that slanted - put a rock face there instead."

Sergio made something on the order of 1800 rules. (No one said he isn't thorough!!) And when we were done, we realized that we barely use landuse.

In developing the rule set, Sergio looked for the parameters that would best predict the real look of the terrain. And what he found was that climate and slope are much better predictors of land use than the actual land use data. If you didn't realize that we were ignoring the input data, well, that speaks to the quality of his rule set.

No One Is Listening

Now back to MeshTool. MeshTool uses the rule set Sergio developed to pick terrain when you have an area tagged as terrain_Natural. If you were to change the land use data, 80% of your land would ignore your markings because the ruleset is based on many other factors besides landuse. Simply put, no one would be listening.

(We could try some experiments with customizing the land use data..there is a very small number of land uses that are keyed into the rule set. My guess is that this would be a very indirect and thus frustrating way to work, e.g. "I said city goes here, why is it not there?")

The Future

I am working with alpilotx - he is producing a next-gen land-use data set, and it's an entirely different world from the raw GLCC that Sergio and I had a few years ago. Alpilotx's data set is high res, extremely accurate, and carefully combined and processed from several modern, high quality sources.

This of course means the rules have to change, and that's the challenge we are looking at now - how much do we trust the new landuse vs. some of the other indicators that proved to be reliable.

Someday MeshTool may use this new landuse data and a new ruleset that follows it. At that point it could make sense to allow MeshTool to accept raster landuse data replacements. But for now I think it would be an exercise in frustration.

12 GB For the RenderFarm

The global scenery gets cut on "the renderfarm", which is our name for the cluster of computers, each loaded with all of the input data for the global scenery. These computers chug for a few days to churn out the whole planet.

With the next global render I am trying an experiment: using one 8-core Mac Pro as the RenderFarm. We've never had more than 8 processors in the farm at a time; in the past to get 5 processors might have taken 3 machines. The appeal of a single machine is ease of setup; no data to sync between machines, no sorting out which machine did which tile and merging it all back.

Today I upgraded the Mac Pro's memory (again) to 12 GB. I thought the logic of why I did this might be of interest to X-Plane users who are trying to figure out "should I have more memory"?

Basically there are three memory limits we care about:

1. The virtual memory limit per process - generally 3 GB per process for a 32-bit application. If an application wants more memory tan this, regardless of what you have, it is dead.
2. The virtual memory limit for the whole machine. Since the machine virtual memory limit is a function of hard disk space, normal users will never care about this - we can have a huge amount of virtual memory.
3. The physical memory actually used by the sum of all programs actually running. Once all programs need more physical memory than you have, they start using hard drive, and they get really, really, really slow.

In the case of the render farm, our processing program runs on one DSF tile at a time. But with 18,000+ tiles we can take advantage of more cores by processing 8 tiles (using 8 copies of the program running at once) on 8 cores.

This is where memory comes in. Before the upgrade my machine had 4 GB of memory, allowing each of the 8 tiles to use a little bit less than 512 MB of RAM before we ran out of physical memory and started paging. (The OS takes a little bit for itself.)

Normally this is all good -- a typical run might only use 300 MB of RAM. But every now and then we hit New York City or Boston and the RAM use spikes out to 1500 MB or more.

This is okay if just one process hits New York, but what if a bunch of processes hit a big city at once? We blow past the physical RAM of the machine and every tile becomes slow at once. And because they are all slow at once, they take a very long time (hours) to clear this state.

(This state of thrash due to many processes is like 8 people trying to go through a doorway at once. If they would just take turns, they'd be through in a second. But by forcing 8 at once they get stuck. The operating system won't pause a few of the high-memory programs to let the others complete.)

Since the RenderFarm has to run overnight unattended, I upgraded RAM to 12 GB, for 1500 MB per rpocess before thrash. My hope is that for this investment, we'll be able to run the processing through without a human to unjam it.

What About X-Plane

So would a RAM upgrade for X-Plane help? Well we can apply what we know from above to figure it out. Generally, you need enough RAM that X-Plane + all other programs won't run out of physical memory. Since X-Plane is 32-bit (and can only use 3 GB of RAM) you are likely to be fine with 4 GB when running X-Plane + the OS. Any more and X-Plane can't take advantage.

The exception might be if you need to run X-Plane and another big program like PhotoShop at the same time. At that point you might want enough RAM for both to run at the same time.

OSM: What You Can Do

I've been working with OpenStreetMap (OSM) data this week. The great thing about OSM (besides already containing a huge amount of road data) is that you can edit and correct data - that is, OSM manages the problem of "crowd sourcing" world map source data.

I get email from people all the time, saying "how can I help fix my local area of the global scenery". With OSM, you can help., by improving OSM's source data.

Here are two things that will matter in the quality of generated scenery:

1. The oneway tag. Roads that are one-way need to have this tag, or the conversion to X-Plane might have an incorrect two-way road in place. If you don't see the one-way arrows on the OSM map rendering then this tag might be missing.
2. The layer tag. When roads cross, "layer" tells OSM which one is on top (and that they do not intersect). Similarly, if a highway is underground, it's because it has a negative layer. If the layer tag is missing, complex intersections will probably render as junk.

In the US, a lot of OSM is built off an import of the TIGER census road map. Unfortunately TIGER in its previous released form does not contain one-way or layering information. So particularly for US cities, adding these tags will improve the road rendering a lot!

ASTER For Custom Scenery

A few days ago, the ASTER GDEM was released. Basically ASTER GDEM is a new elevation set with even greater coverage than the SRTM. Basically both SRTM and ASTER (I'll drop the GDEM - in fact ASTER prodcues more than just elevation data, but the elevation data is what gets flight simmers excited right now) are space-based automated measurement of the earth's height. But since ASTER is on a satellite (as opposed to an orbiting space shuttle) it can reach latitudes closer to the poles.

So what does this mean for scenery? What does it mean for the global scenery? A few thoughts:

• ASTER data is not yet very easy to get. You can sign up with the USGS distribution website but you're limited to 100 tiles at a time, with some latency between when you ask and when you get an FTP site. Compare this to SRTM, which can be downloaded automatically in its entirety, or ordered on DVD. ASTER may reach this level of availability, but it's not there yet.

• ASTER is, well, lumpy. (Nasa says "research grade", but you and I can say "lumpy".) Jonathan de Ferranti describes ASTER and its limitations in quite some detail. Of particular note is that while the file resolution is 30m, the effective resolution of useful data will be less.

  SRTM has its defects, too, but ASTER is very new, so the GIS community hasn't had a chance to produce "cleaned up" ASTER. And clean-up matters; it only takes one really nice big spike in a flat flood plane to make a "bug" in global scenery. I grabbed the ASTER DEMs for the Grand Canyon. Coverage was quite good, despite the steep terrain angle (steep terrain is problematic by design for SRTM) but there were still drop-out areas that were filled with SRTM3 DEMs, and the filled-in area was noticeable.

• By the numbers, ASTER is not as good as NED; I imagine that other country-specific national elevation datasets are also both more accurate and more precise than ASTER.

• The licensing terms are, well, unclear. The agreements I've seen imply a limited set of research uses for the data. The copyright terms are not well specified.
  

So at this point I think ASTER is a great new resource for custom scenery, where an author can grab an ASTER DEM in a reasonable amount of time, check it carefully, and thus have access to high quality data for remote parts of the Earth, particularly areas where locally grown data is not available or not high quality.

In the long term, ASTER is a huge addition to the set of data available because of its wide-scale coverage of remote areas, and because it can fill holes in SRTM. (ASTER and SRTM suffer from different causes for drop-outs, so it is imaginable that there won't be a 1:1 correlation in drop-outs.)

But in the short term, I don't think ASTER is a SRTM replacement for global scenery; void-filled SRTM is a mature product, reasonably free of weirdness (and sometimes useful data). ASTER is very new, and exciting, but not ready for use in global scenery.

This Is Where SRTM Comes From

My family visited DC this weekend and we went out to Udvar-Hazy, the extension of the Smithsonian aerospace museum out near Dulles international airport. My dad took this picture.

That is one of the two radar antennas (and the telescoping arm) used to scan the earth as part of the Shuttle Radar Topography Mission (SRTM). The SRTM is basically the first really good quality most-of-the-earth elevation dataset, and it is the main (but not only) source of elevation data for the X-Plane global scenery.

The telescoping mast shown in the picture (horizontal) extends one of the two radar antennas away from the shuttle when in orbit; had they not been able to retract the antenna they would have had to detach it and leave it in space. Fortunately the mechanism worked properly, so they were able to bring the antenna back for posterity.

To Wiki or Not To Wiki

You might not believe this (due to the general lack of scenery system documentation) but I do spend some brain power thinking about X-Plane documentation for third parties!

Consider two approaches to documentation:

• http://scenery.x-plane.com/
• http://wiki.x-plane.com/

My question is: which of these approaches is more "readable" or "clear" to you as a third party? Each one (the formal website vs. the Wiki) has pros and cons, but I can't judge "usability" of the documentation myself.  Is it easier to find things on the website?  On the Wiki?  Comments welcome!

(I need to decide where to put future documentation, hence the question "which works better for those who read the documentation.)

Auto-Variation and Repetition

In my previous post I discussed variation as a way to hide the artifacts of land use texturing. Now we can talk about this bug.

What are these weird artifacts that show up over the terrain when shaders are on?  Well, they should (and will in 930) look like this:

But what's going on?  The answer is auto-variation.

In X-Plane 8, variation is created by using multiple layers, each one applying a texture at a different offset.  This technique works on a wide range of hardware, but is inefficient - it causes overdraw (which we know is very bad).  

So in X-Plane 9 I replaced this layer-based variation with a pixel shader algorithm.  This means less information in the DSF (which means smaller DSFs, faster loading and less RAM use), but it also means variation is only visible to those with shaders.  Having the pixel shaders create variation dynamically on the GPU is called "auto-variation" (and is invoked via the AUTO_VARY command in a .ter file).

The artifact above was a bug in the auto-variation shader.  With the code now fixed (the 930 patch will contain the fix), here are some images of how it is supposed to work:

Here we have the texture in question, at two different offsets.


This black and white texture is the "mixing mask" used to select which offset to use.

And this is the final result.

There is a little bit more disruption in the columns of green park.

Dealing With Repetition

I was going to post some pictures of the newly fixed "auto-vary" feature, but before I can do that in a way that makes any sense, I need to explain how X-Plane deals with texture repetition.

Texture repetition is the inevitable result of using "landuse-style" texturing (that is, a repeating single texture representing a type of land).  Typical X-Plane land use textures are 1024 x 1024 at max res and repeat about every 3-5 km.  Unfortunately, our brains are pattern-recognizing machines, and the result of this texturing scheme is that the "grid lines" of texture placement become apparent over wide views.  

We use a number of techniques to minimize this problem.

Lots of Land Uses

Our main tool to combat repetition is to not use a given land use for too large of an area.  This has the advantage of efficiently using the entire set of textures, and (because terrain textures change based on an irregular grid, based on elevation) the changes to textures are both irregular in shape and "plausible" in placement.

In this picture, you can see that the urban residential land use has been interrupted by various forest and grass textures.  This is intentional - Sergio tuned hte land use rules to make sure that we wouldn't have large regions of one land use type.  Those blobs match the irregular grid, which gets its shape from the terrain's elevation.

Variation

In the above picture, you can still see the repeating grid of the residential terrain; observe the right side - you'll see the same repeating vertical pattern of road repeating over and over.  In order to further hide repetition, we use the same texture multiple times, but in offset locations.

Here you can see that the vertical line on the right side has been broken up a bit.

More VRAM

In some cases, a terrain covers such large areas despite the rule set (e.g. for really flat areas) that we use two separate textures and can vary between them.  Here you can see both input textures for our dry square crop land use, as well as the combined results.

In summary, we have three techniques:

1. Add more rules to prevent large spans of a single land-use.
2. Use a texture with multiple offsets (variation)
3. Use two textures and vary between them.

What Does the Airport Boundary Do

The apt.dat 850 file format defines a polygonal "airport boundary". But what exactly does it do? It does different things when creating DSFs and when rendering them.

Inside X-Plane it has relatively little effect:

• It is one of many elements that counts toward the land area that will be flattened. (Runways and taxiways are also used.)
• We do not actually render any special terrain or fences.

The airport boundary has more of an effect during scenery creation.

• If an airport has an airport boundary, we do not calculate the airport's boundary ourselves - instead f we use the specified boundary.
• All land intersecting the airport boundary is turned to airport terrain.
• The DEM is flattened within the airport boundary to reduce the slope of high frequency bumps.

There is one aspect of airport creation that the airport boundary is not involved in: filling in water to make land under runways. When we create an airport, we actually calculate three boundary polygons:

• The inner ring is closest to runways and taxiways (very close) with very little simplification. It is filled in with land if it is wet.
• A second ring slightly farther from runways and taxiways is also filled in with land if it is wet.
• The outermost ring is a lot farther out, but does not fill in Water. This is the ring that the airport boundary can replace.

Why do we need two inner rings? Well, if an airport is next to the water but not at sea level, we need to induce two sets of mesh points, the outer ones which drop down to sea level and the inner ones which are at airport level. You can see the importance of this at KLGA, where one of the runways dropped to sea level at its midpoint in version 8, but not version 9.

Since the airport boundary polygon provides only one ring, it cannot be used for this purpose. At some point in the future, we might use the airport boundary for rings 2 and 3, or use a smaller version of the airport polygon for rings 1 and 2.

For now, my recommendation is: the airport boundary should trace out the entire airport premesis, not including water.

Precomputed Scenery - the Good and the Bad

This thread on X-Plane.org sparked off quite the discussion. Now a lot of this is a discussion of when LR will have an overlay editor - there are a few overlay editing functions that Jonathan Harris' excellent OverlayEditor apparently does not yet support, sparking this discussion.

(I am not saying that LR should rely on Jonathan to do an overlay editor. But I am saying that the complaints I hear about a lack of overlay editing go down when Jonathan's overlay editor does everything that the file formats can do.)

But another part of the discussion focused on the problem of mesh editing. In particular, the basic terrain in a DSF is a fully baked output of a complex process that starts with higher level GIS input data. In other words, we start with a raster DEM, polygon coastline, apt.dat file, vector roads, and a bunch of config files and hit "bake" and a DSF comes out the other side, with a lot of processing.

This is very different than FS X, which integrates its data sources on the fly. Why did we choose a precomputed route for scenery? It has some pros and cons. (In understanding how we made these decisions, think back to what scenery was like with X-Plane 7 and ENVs and single-core machines.)

Performance

The main benefits of preprocessing scenery are performance related. When you process scenery data into the final scenery while flying, that computer power takes away from the rendering engine, thus cutting down fps. At some point you have a zero-sum game between how much cost there is to loading scenery and how complex the scenery integration can be; you have to pick very simple scenery integration algorithms to keep fps up.

(This is less of an issue as more cores become available, but is still a factor.)

When pre-processing, we can use algorithms that take minutes per DSF without affecting framerate.

Similarly, there might be scenery processing algorithms that improve fps by optimizing the output triangles - but do we have time to run these algorithms during load? With preprocessing we have all the time in the world because it happens once before the DVDs are burned.

Preprocessing also breaks a similar zero sum game between scenery data size and quality; the source data we use to make the scenery is a lot bigger than the 78 GB of DSFs we cut; if we had to ship the source data, we'd have to cut down the source data quality to hit our DVD limitations. With be-baking we could use 500 GB of source data without penalty.

Format Flexibility and Stability

The second set of benefits to preprocessing are flexibility benefits. (Consider the file format churn of the ENV days.)

- With a preprocessed scenery file, what the author creates is what the user sees - X-Plane does not go in and perform subjective integrations on the scenery later that might change how it looks in a negative way.

• There is no need to revise the scenery file formats to introduce new data sets, because new data sets and old are all processed to the same final DSF container format.
• A wide variety of mesh generation techniques can be employed because the mesh generation is not built into X-Plane. This is a flexibility that I don't think anyone has really utilized.
• Changes of behavior in the scenery generation toolset can never affect existing scenery because that scenery is already preprocessed; this help compatibility of old file formats.

Integration Issues

There are some real limitations to a pre-processed format, and they are virtually all in the bucket of "integration issues" - that is, combining separate third party add-ons to improve scenery. In particular, in any case where we preprocess two data sources, we lose the opportunity for third parties to provide new scenery to replace one of those data sources and not the other.

Airport is the achilles heal where this hurts us most; while airport layouts are overlays and can be added separately to the scenery system, the elevation of the base mesh below the airport needs to be preprocessed. This is something I am still investigating - a tolerable fix that other shave proposed is to allow an overlay scenery pack to flatten a specific region regardless of the user setting (so an author can be assured of a flat base to work from).

Preprocessing does fundamentally limit the types of third party add-ons that can be done; with version 9 and overlay roads, we are getting closer to letting road add-ons be overlays (see this post).

It appears to me that integration isn't the primary complaint about the scenery system (the primary complaint is lack of tools) but we'll have to see once we have mesh editing tools (mesh recreation tools really) whether preprocessing still limits certain kinds of scenery.

Note that a lack of tools or a lack of tool capability is not an inherent limitation of pre-processed scenery. We have an incomplete tool set because I have not written the code for a complete tool set, not because it cannot be done.

(The complexity of writing base mesh editing tools is a function of the complexity of a vector-based base mesh - this is also not related to pre-processing per se.)

Tools

In the end, I think the question of tools is not directly tied to the question of pre-processing. Whether we have scenery that is processed by X-Plane or a preprocessing tool, we have the same issues:

• Good tools require an investment in coding user interface.
• The code to convert source data which users might want to edit (like a polygon that defines a lake) to data the simulator might want to use (like a list of 78,231 triangles) has to be written.

I don't think either option (pre-processing or in-simulator processing) reduces the amount of work to be done to create a good toolset.

As a final thought, using scenery file formats that are "easier to edit" (e.g. a file format that contains a polygon for water rather than triangles) doesn't make the total code for scenery tools + simulator any easier; it just moves the task of "processing" the scenery from the tools to the simulator itself.

The Future of Triangles Part 4: Pie in the Sky

Per-pixel lighting is something I hope to have in X-Plane soon.  A number of other features will take longer, and quite possibly might never happen.  This is the "pie in the sky" list - with this list, we're looking at higher hardware requirements, a lot of development time, and potential fundamental problems in the rendering algorithm!

High Dynamic Range (HDR) Lighting

HDR is a process whereby a program renders its scene with super bright and super dark regions, using a more detailed frame-buffer to draw.  When it comes time to show the image, some kind of "mapping" algorithm then represents that image using the limited contrast available on a computer monitor.  Typical approaches include:

• Scaling the brightness of the scene to mimic what our eyes do in dark or bright scenes.
• Creating "bloom", or blown out white regions, around very bright areas.

Besides creating more plausible lighting, the mathematics behind an HDR render would also potentially improve the look of lit textures when they are far away.  (Right now, a lit and dark pixel are blended to make semi-lit pixels when far away as the texture scales down.  If a lit pixel can be "super-bright" it will still look bright even after such blending.)

Besides development time, HDR requires serious hardware; the process of drawing to a framebuffer with the range to draw chews up a lot of GPU power, so HDR would be appropriate for a card like the GeForce 8800.

While there aren't any technical hurdles to stop us from implementing HDR, I must point out that, given a number of the "art" features of X-Plane like the sun glare, HDR might not be as noticeable as you'd think.  For example, our sun "glares" when you look at it (similar to an HDR trick), but this is done simply by us detecting the view angle and drawing the glare in.

Reflection Mapped Airplanes

Reflection maps are textures of the environment that are mapped onto the airplane to create the appearance of a shiny reflective surface.  We already have one reflection map: the sky and possibly scenery are mapped onto the water to create water reflections.

Reflection maps are very much possible, but they are also very expensive; we have to go through a drawing pass to prepare each one.  And reflection maps for 3-d objects like airplanes usually have to be done via cube maps, which means six environment maps!

There's a lot of room for cheating when it comes to environment maps.  For example: rendering environment maps with pre-made images or with simplified worlds.

Shadows

Shadows are the biggest missing feature in the sim's rendering path, and they are also by far the hardest to code.  I always hesitate to announce any in-progress code because there is a risk it won't work.  But in this case I can do so safely:

I have already coded global shadow maps, and we are not going to enable it in X-Plane.  The technique just doesn't work.  The code has been ripped out and I am going to have to try again with a different approach.

The problem with shadows is the combination of two unfortunate facts:

• The X-Plane world is very, very big and
• The human eye is very, very picky when it comes to shadows.

For reflections, we can cheat a lot -- if we don't get something quite right, the water waves hide a lot of sins.  (To work on the water, I have to turn the waves completely off to see what I' m doing!)  By comparison, anything less than perfect shadows really sticks out.

Shadow maps fail for X-Plane because it's a technology with limited resolution in a very large world.  At best I could apply shadows to the nearest 500 - 1000 meters, which is nice for an airport, but still pretty useless for most situations.

(Lest someone send the paper to me, I already tried "TSM" - X-Plane is off by about a factor of 10 in shadow map res; TSM gives us about 50% better texture use, which isn't even close.)

A user mentioned stencil shadow volumes, which would be an alternative to shadow maps.  I don't think they're viable for X-Plane; stencil shadow volumes require regenerating the shadow volumes any time the relative orientation of the shadow caster and the light source change; for a plane in flight this is every single plane.  Given the complexity of planes that are being created, I believe that they would perform even worse than shadow maps; where shadow maps run out of resolution, stencil shadow volumes would bury the CPU and PCIe bus with per-frame geometry.  Stencil shadow volumes also have the problem of not shadowing correctly for alpha-based transparent geometry.

(Theoretically geometry shaders could be used to generate stencil shadow volumes; in practice, geometry shaders have their own performance/throughput limitations - see below for more.)

Shadows matter a lot, and I am sure I will burn a lot more of my developer time working on them.  But I can also say that they're about the hardest rendering problem I'm looking at.

Dynamic Tessellation

Finally, I've spent some time looking at graphics-card based tessellation.  This is a process whereby the graphics card splits triangles into more triangles to make curved surfaces look more round.  The advantage of this would be lower triangle counts - the graphics card can split only the triangles that are close to the foreground for super-round surfaces.

The problem with dynamic tessellation is that the performance of the hardware is not yet that good.  I tried implementing tessellation using geometry shaders, and the performance is poor enough that you'd be better off simply using more triangles (which is what everyone does now).

I still have hopes for this; ATI's Radeon HD cards have a hardware tessellator and from what I've heard its performance is very good.  If this kind of functionality ends up in the DirectX 11 specification, we'll see comparable hardware on nVidia's side and an OpenGL extension.

(I will comment more on this later, but: X-Plane does not use DirectX - we use OpenGL.  We have no plans to switch from OpenGL to DirectX, or to drop support for Linux or the Mac.  Do not panic!  I mention DirectX 11 only because ATI and nVidia pay attention to the DirectX specification and thus functionality in DirectX tends to be functionality that is available on all modern cards.  We will use new features when they are available via OpenGL drivers, which usually happens within a few months of the cards being released, if not sooner.)

Why We Must Edit the Source Data

My previous post on scenery tools stirred up some discussion; y-man brought up a point fundamental enough that I think it warrants explanation.

The questions is whether to fix broken DSFs by editing the source data or the DSFs themselves.

Let me be clear: both are viable options, both have limitations, and neither are possible today. So in choosing one over the other, I am picking a strategy that I think is superior, and discounting the other not because I think it is useless, but because it is less useful and development time is very limited (doing both would take away from other good features).

Basically the two ways to fix a DSF are:

1. Edit the DSF itself until it looks correct.
2. Edit the source data and then rebuild the DSF from scratch.

I strongly believe we must pursue the second strategy for this simple reason: if we correct a DSF but don't fix the source data, the same mistakes will be made in the future.

We have to keep recutting the global scenery to keep up with:

• Higher capacities for detail in newer computers.
• New global data that becomes available.
• Improved generation algorithms that makes better results from existing data.

To lose any of these would be a big set-back in scenery quality, so not recutting DSFs isn't a great option. (Furthermore, improvements in scenery often come from new global data, so picking user changes over new data would be a tough choice.)

By letting users change the source data, we can have the best of both worlds: problems are fixed while new technology is adopted.

To answer y-man's direct questions:

But that data is not available to us users is it? If it is, where is it, and where is the spec to work with it?

It is not available yet! I am proposing to focus on making the data available and creating the infrastructure to share data and receive improvements (choice 2) rather than providing a DSF mesh editor (choice 1).

Alternatively, a user who goes thru the trouble of correcting base DSF could send in the modified DSF, or may be even a diff between before and after states of DSF text files, and attach an explanation of purpose.

Here's the problem: we can't preserve the diff to the DSF and apply it to future renderings.

Imagine, or example, that you relocate 500 mesh vertices from the existing DSFs to correct a coastline. (I am being generous here and assuming a clear, single, unifying edit. But some authors would more likely move the vertices to make a number of improvements.)

In the meantime, another author creates an airport nearby, and someone else improves the SRTMs (there is at least one person attributed in our about box who has been collecting void-filled SRTM tiles). The effect of the nearby airport's pavement changing size and the raw elevation changing height is to change greatly how the mesh is generated, such that 400 of the 500 vertices that were moved simply no longer exist, and 450 new vertices are in the nearby area now that were not in the original DSF.

This case is known as a "merge conflict" in computer programming terms (and happens when two changes to a program are "merged"). The problem is that we can't sanely know what to do with our 500 edited vertices in this case. Do we take the 100 vertices that still exist and move them without the others? What if that produces very strange results? (Triangles might become inside-out because some vertices are moved a lot but their neighbors are not because they did not match the diff.)

We could try to apply some kind of change to the new vertices similar to what happened to the old, but how do we know if this is making things better or worse? What if that change simply deforms the outline of the airport that was added?

I can go on and on with these kinds of examples, but the point is this: you can't unbake a cake. A DSF is similar; you can't necessarily recover the sources that were processed together to form the final product.

This is why it's important that we create infrastructure to correct source data, rather than focus on editing the final DSFs. I can assure you that my negative attitude toward editing DSFs is not a negative attitude toward user participation!

There are still a lot of details to be worked out about how we can work together. Who will own the data, and on what terms? How will it be distributed? How will it be shared?

Unfortunately I can't answer those questions yet. I still need to do more research into what is technologically possible - then we can figure out how to proceed.

The Future of WED

WED 1.0 has gone RC. The only change from beta 5 is that I have the latest manual changes from Tom (including Cormac's illustrations of taxiway signs). Both of them did some great work - the WED manual is a lot nicer than the docs I have done myself for the other tools.

A while ago I posted a tools update...has anything changed? Not a whole lot.

• WED is definitely the future of the "heavy" scenery tools (ones with extensive UI) - it has a lot of infrastructure for features like multiple undo, multiple selection, hiearchial editing, etc.
  
• The next big feature for WED will be some kind of overlay editing. I don't think this will happen very soon though - my todo list is pretty out of control.
• In the long term, I think WED may provide a visual way to do MeshTool-like operations. In other words, you'll be able to build a base mesh in WED by specifying polygons for airports, applying orthophotos, and importing one big DEM per tile.
• I don't think that WED will ever be a DSF editor - that is, you won't be able to open an existing DSF, move a single node, and re-save it. This just isn't high on my priority list.

To this last point, how are you supposed to fix scenery if you can't edit DSFs? Well, I'll try to post more on that over the next few weeks, but the short answer is that we need structured edits to source data.

A DSF is a compiled result of a very complex set of processes. The vertex that you adjust in the mesh to fix a mountain top may not exist in the DSF if another user submits an updated airport layout, even if they are fifty kilometers apart. (All parts of the DSF affect each other through the adaptive irregular mesh.) So I don't want to take user submitted corrections to the final DSF because those changes would be lost at the next render.

We've been down this road before - when the V6 global scenery was done, the plan was: now we'll take user-edited ENVs. The problem was that this plan assumed that we'd never re-render the ENVs again, which proved totally wrong - we re-rendered them at the end of the v6 run with improved algorithms.

What we need to do is identify what components of the source data are reasonable candidates for user editing, and set up processes (similar to Robin's collection of apt.dat data) to gather data and share it.

(We are looking at OSM - it is still under investigation!)

X-Plane 9 Is Here

There will be a lot more to post about this soon...a few immediate comments:

We have an order in with one of our server providers to get a lot more bandwidth for demo/update downloading...the servers are going to be pretty slammed, but we're doing our best to keep downloads fast. I believe the beta should be about 600 - 700 MB over the net.

X-Plane beta 1 is on the DVD. X-Plane beta 2 is on the net. In other words, we've already done our first patch since the DVDs went master. This patch improves performance and fixes QuickTime movie recording on Windows. (BTW QT movie recording is completely overhauled for version 9.) So once you get DVDs, make sure to run the web updater to get beta 2. With a DVD install this is a tiny download, since the updater only fetches changed files.

The DVD set is "X-Plane 9 beta 1", but the scenery and terrain/earth orbit textures are final. So you can buy DVDs now and simply use the web updater to get the "final" X-Plane 9.00 (plus of course all of the free upgrades and patches for the life of version 9).

This is the first major version release where we've had our updater. In the past we've had to either post a new version without a beta, or put a beta on DVD. With X-Plane 9, the large files that aren't available for web updating are finished, and the rest of the beta can be upgraded over the net. I hope this allows people to participate in the beta program and still enjoy the sim with full scenery.