Saturday, May 12, 2007

You Can't Unbake a Cake

First, I just want to be clear: I am not announcing any future features for the scenery tools. I am not saying when they will be released, and I am not saying what they will do, because honestly I do not know. There have been too many cases when users have emailed me and said either "I thought you guys were going to do X" or "you guys promised you were going to do X", so now I am officially paranoid.

So this blog post is not about what the future scenery tools will do - it is simply a discussion of the difference between editing source data and compiled DSFs.

When we make the global scenery, we start with a bunch of source data that roughly consists of: road maps, coastlines, elevation, landuse, climate data, airport locations, etc. When we build a DSF out of it, we "bake" these items together into a single file. During this baking process, our tools apply some "integration effects". Here are a few of the more obvious examples:
  • Terrain under airports is forced to be an airport grass texture, appropriate for local climate data.
  • Roads are removed from airport areas.
  • Intersections are computed for highways - that is, a highway and city street form an overpass, but two city streets become a real intersection.
  • Generated Buildings are put around the roads, not under them.
  • Buildings are oriented to "face" the slope of the ground they are under, based on their shape.
That's not a complete list, but it gives you an idea of some of what goes into making a DSF. All of this information is precomputed and represented in the final DSF. But consider this last point: the DSF contains the actual orientation (north/south/east/west) of each building. It does not contain the slope underneath the building and it contains no information about which buildings need reorientation. In other words, the results of the process, not the inputs to the process, are present.

So consider what would happen if you could simply edit the data in the DSF:
  • If you moved an airport, the airport grass would stay in its own location.
  • If you moved an airport over a road, the road would still be there.
  • If you changed a city street to a highway, it would not form an overpass.
  • If you moved a road on top of a generated building, the building would remain in place.
  • If you changed the ground elevation, buildings would not change their orientation to face the new slope.
If you changed the source data and re-ran the DSF building process, these effects would occur. But remember, we need elevation, land use, etc. to build a DSF from scratch, and the DSF itself doesn't contain its source data.

So we have two possible strategies for editing DSFs:
  1. We could build a DSF "retouching" tool that let us make very small changes to the existing DSF without having any source data. None of these effects would "work" so authors would have to make very small changes and then hand-fix any problems that appeared.
  2. We could build a DSF "rebuilding" tool that let authors make new DSFs from source data. All of the effects would look good, but authors would have to get some of the source data. (We could post our source data, or provide links to places where it can be downloaded.)
Note that we can't have our cake and eat it...we cannot get the "integration effects" listed above unless we go back to the source data. If I can stretch my cake metaphor to the breaking point, once we make our cake, we can't easily remove the flour and add another egg - we need to start over with the raw ingredients.

Which strategy will the scenery tools use? I don't know yet. I am focusing on airport and overlay editing, which sidestep this issue a bit (we can easily edit an apt.dat 850 or overlay from the final product). We may do a bit of both strategies - it depends on what users want and what we can code efficiently.

Why don't the finished DSFs contain everything we need to edit them? The answer is size. The finished global scenery was about 56 GB of DSF. When I last checked, the raw data that forms the DSFs was at least 100 GB or more. So for each DVD we shipped of scenery, we'd have to ship two more DVDs of source data, for a 21 DVD set, most of which wouldn't be useful to most users.


Austin said...

Interesting. It sounds like may have to build in a tool to automatically download source data for a region before the user can start to edit it. It would seem sensible to encapsulate all the source data into the XES file. How about pre-building XES files for the world, using the same source data you used for the original v8 worldwide scenery, and including a module in WED to automatically download the XES file for the region the user wants to work on. This avoids a 100Gb download when most users will only want to edit one scenery tile anyway?

andras.fabian said...

I would vouch for the "rebuild from raw data" way. And of course make it as open as possible for the raw data (at least to an extent). Of course, you shouldn't integrate all the features of a full fledged GIS tool - no need to reinvent the wheel. For data preparation we can still use those tools. But the open input process would make it possible, to use other data than you used (mostly, more detailed, or better in any other way). I'm really looking forward to such a tool ;-)))

Andras Fabian / alpilotx

Anonymous said...

As I said on the related thread on .org, I think most users are looking for option 1, a tweaking tool, while option 2 would have the potential of allowing some power users to do some really cool stuff. In that respect, I really think LR should have the (not too) long term goal of implementing both strategies with the priority given to option 1 in order to satisfy most users first.

- Cormac

Anonymous said...

I see some arguments against being able to edit existing DSFs which are not very strong IMHO. For the most part, users want to be able to touch up the existing Global scenery while adding more detailed airports which for the near term is adequate. Here are the arguments listed and some rebuttals.

* If you moved an airport, the airport grass would stay in its own location.

This editing feature would seem to be the toughest to implement properly due to the terrain mesh needing re-textured. When I first encountered this, I was actually quite surprised that the decision was made to hard code the airport grassy areas into the DSF terrain mesh. The later developed draped polygon feature with a seperate polygonal boundry for a flattening code would have been a much better option though I realize that it was not available at the time. It would be nice to see this changed in the next round (ie v9). That said, the majority of the large airports are fine in this regard. Albeit, there are smaller airports that are drastically wrong. I think most people can live with a bit of extra grass here and there.

* If you moved an airport over a road, the road would still be there.

The ability to create, delete, edit and move roads is really a must have option. Any thoughts of not having this feature is not viable. Therefore, if an author did move an airport, he would be able to see this conflict and move/delete the road. This makes this somewhat of a non-issue.

* If you changed a city street to a highway, it would not form an overpass.

This is where the road editing feature as well as most other features would really gain from a proofing code that checks for these things in the areas that have been edited. But as with the airport, the scenery creator should be able to easily edit the road.

* If you moved a road on top of a generated building, the building would remain in place.

Same as above with the exception that proofing code to check if a building is placed on a road would be a nice option.

* If you changed the ground elevation, buildings would not change their orientation to face the new slope.

As above, the scenery creator should be able to create/delete/move buildings with WED. This, like road editing, is also a must have in reality.


It seems that the tack that is being taken is to steer scenery creators toward scratch built DSFs. I can understand if this is the case. Editing full DSF's is a royal PITA due to the amount of data involved. For the most part, the Global Scenery terrain is great and just needs touching up (ie move/change/add/delete a building/road here or there). If editing requires a scratch build, a lot of potential users are going to be put off from creating scenery as it will just take far too much time for them.


Benjamin Supnik said...

Hi Voynik,

You make some good points here, and I think you are right that the issues with "integration effects" do NOT prohibit the editing of existing DSFs...the point I am trying to make is only that it is more tedious (relative to editing the source data) because a large number of small elements must be manually correlated (or I must rewrite into the tool some "reintegration" code).

Regarding the airport areas: two thoughts:
- You are right that draped polygons did not exist in v8.0, heck they didn't exist until version 850. They also depend on the asynchronous threaded build-up of 3-d scenery, something that is much more viable now that dualcore chips have become the rage.

- Even if we did use overlays for all airport areas, the most important effect of the airport creation and prep code is not just that it changes the texture, but that it changes the mesh (flattening it) in a manner that is much more sophisticated than x-plane can do, and this flattening has an effect on the shape of the triangles (generally making them bigger) that helps the overlay polygon code run efficiently.

So I think we must do a lot of preprocessing to airports even if we use .pol files for the surface areas...given this and the general tendency of airports not to move after they're built, I suspect we'll continue with the existing strategy and continue to bake airports into the terrrain in the next major render.

Remember that .pols are always overlays - thereh as to be something under them. I don't see any harm in that thing being airport grass. :-)