It's that time of the year again...I'll be out of the office next week, and more importantly, away from all cell and email contact.
Austin and Randy will be at EAA - stop by and say hi if you're there. Once Austin gets back, the last few betas should resume - we're close to an RC on 920 but not quite there.
Friday, July 25, 2008
Monday, July 21, 2008
Report It, Don't Fix It
I just found two bugs in the scenery system in beta 7 (both will be fixed in beta 8):
My goal with scenery is to avoid regression bugs - that is, if it works in 900, it should work in 920. So I was glad to get these reports. I cringe when I read posts where people report "fixing" problems with content that are new to 920. Those are my bugs to fix!
- If an object had a blank TEXTURE statement with no trailing white space (this is legal in OBJ8, at least I think) the sim would crash when it rendered the object.
- DDS-based terrain textures with the new paging system would show junk when they were far away.
My goal with scenery is to avoid regression bugs - that is, if it works in 900, it should work in 920. So I was glad to get these reports. I cringe when I read posts where people report "fixing" problems with content that are new to 920. Those are my bugs to fix!
Thursday, July 17, 2008
Why We Must Edit the Source Data
My previous post on scenery tools stirred up some discussion; y-man brought up a point fundamental enough that I think it warrants explanation.
The questions is whether to fix broken DSFs by editing the source data or the DSFs themselves.
Let me be clear: both are viable options, both have limitations, and neither are possible today. So in choosing one over the other, I am picking a strategy that I think is superior, and discounting the other not because I think it is useless, but because it is less useful and development time is very limited (doing both would take away from other good features).
Basically the two ways to fix a DSF are:
We have to keep recutting the global scenery to keep up with:
By letting users change the source data, we can have the best of both worlds: problems are fixed while new technology is adopted.
To answer y-man's direct questions:
Imagine, or example, that you relocate 500 mesh vertices from the existing DSFs to correct a coastline. (I am being generous here and assuming a clear, single, unifying edit. But some authors would more likely move the vertices to make a number of improvements.)
In the meantime, another author creates an airport nearby, and someone else improves the SRTMs (there is at least one person attributed in our about box who has been collecting void-filled SRTM tiles). The effect of the nearby airport's pavement changing size and the raw elevation changing height is to change greatly how the mesh is generated, such that 400 of the 500 vertices that were moved simply no longer exist, and 450 new vertices are in the nearby area now that were not in the original DSF.
This case is known as a "merge conflict" in computer programming terms (and happens when two changes to a program are "merged"). The problem is that we can't sanely know what to do with our 500 edited vertices in this case. Do we take the 100 vertices that still exist and move them without the others? What if that produces very strange results? (Triangles might become inside-out because some vertices are moved a lot but their neighbors are not because they did not match the diff.)
We could try to apply some kind of change to the new vertices similar to what happened to the old, but how do we know if this is making things better or worse? What if that change simply deforms the outline of the airport that was added?
I can go on and on with these kinds of examples, but the point is this: you can't unbake a cake. A DSF is similar; you can't necessarily recover the sources that were processed together to form the final product.
This is why it's important that we create infrastructure to correct source data, rather than focus on editing the final DSFs. I can assure you that my negative attitude toward editing DSFs is not a negative attitude toward user participation!
There are still a lot of details to be worked out about how we can work together. Who will own the data, and on what terms? How will it be distributed? How will it be shared?
Unfortunately I can't answer those questions yet. I still need to do more research into what is technologically possible - then we can figure out how to proceed.
The questions is whether to fix broken DSFs by editing the source data or the DSFs themselves.
Let me be clear: both are viable options, both have limitations, and neither are possible today. So in choosing one over the other, I am picking a strategy that I think is superior, and discounting the other not because I think it is useless, but because it is less useful and development time is very limited (doing both would take away from other good features).
Basically the two ways to fix a DSF are:
- Edit the DSF itself until it looks correct.
- Edit the source data and then rebuild the DSF from scratch.
We have to keep recutting the global scenery to keep up with:
- Higher capacities for detail in newer computers.
- New global data that becomes available.
- Improved generation algorithms that makes better results from existing data.
By letting users change the source data, we can have the best of both worlds: problems are fixed while new technology is adopted.
To answer y-man's direct questions:
But that data is not available to us users is it? If it is, where is it, and where is the spec to work with it?It is not available yet! I am proposing to focus on making the data available and creating the infrastructure to share data and receive improvements (choice 2) rather than providing a DSF mesh editor (choice 1).
Alternatively, a user who goes thru the trouble of correcting base DSF could send in the modified DSF, or may be even a diff between before and after states of DSF text files, and attach an explanation of purpose.Here's the problem: we can't preserve the diff to the DSF and apply it to future renderings.
Imagine, or example, that you relocate 500 mesh vertices from the existing DSFs to correct a coastline. (I am being generous here and assuming a clear, single, unifying edit. But some authors would more likely move the vertices to make a number of improvements.)
In the meantime, another author creates an airport nearby, and someone else improves the SRTMs (there is at least one person attributed in our about box who has been collecting void-filled SRTM tiles). The effect of the nearby airport's pavement changing size and the raw elevation changing height is to change greatly how the mesh is generated, such that 400 of the 500 vertices that were moved simply no longer exist, and 450 new vertices are in the nearby area now that were not in the original DSF.
This case is known as a "merge conflict" in computer programming terms (and happens when two changes to a program are "merged"). The problem is that we can't sanely know what to do with our 500 edited vertices in this case. Do we take the 100 vertices that still exist and move them without the others? What if that produces very strange results? (Triangles might become inside-out because some vertices are moved a lot but their neighbors are not because they did not match the diff.)
We could try to apply some kind of change to the new vertices similar to what happened to the old, but how do we know if this is making things better or worse? What if that change simply deforms the outline of the airport that was added?
I can go on and on with these kinds of examples, but the point is this: you can't unbake a cake. A DSF is similar; you can't necessarily recover the sources that were processed together to form the final product.
This is why it's important that we create infrastructure to correct source data, rather than focus on editing the final DSFs. I can assure you that my negative attitude toward editing DSFs is not a negative attitude toward user participation!
There are still a lot of details to be worked out about how we can work together. Who will own the data, and on what terms? How will it be distributed? How will it be shared?
Unfortunately I can't answer those questions yet. I still need to do more research into what is technologically possible - then we can figure out how to proceed.
Monday, July 14, 2008
HUD Hell
For X-Plane 920, I modified the HUD code to draw the HUD into the 3-d cockpit's panel texture. This turned out not to be such a good idea. The problem is one of opting in: if we change the behavior of the sim without authors being able to "opt in" to that new behavior, we break old planes.
The first plane we've seen with this problem is the F-22. The F-22 uses a 3-d object for the cockpit all the time (even in 2-d), and uses the 2-d panel simply for texturing. (In this sense, the F-22 is ahead of its time - this structure gives some of the advantages of the new 3-d panel texture).
The problem is that the F-22 uses the panel texture to map the 2-d HUD to the 3-d HUD, with the expectation that only the HUD background (and not the writing) will appear. The sim then draws the HUD on top. When we start burning the HUD into the panel texture, we get two HUDs.
To fix this, we won't draw the HUD to the panel texture if you have a real 3-d cockpit object. This change will go into beta six or seven.
I may eliminate the HUD on the default 777 as well...the problems with this feature are numerous:
- As discussed above, for any plane with an existing 3-d object, adding the HUD is more likely to break the plane than add value..authors could not plan for this.
- For a plane with no cockpit object like the 777 the HUD may not be very well aligned with anything - it's just a 2-d drawing.
- Even if the HUD texture is useful, because the HUD is drawn to a texture, the result is blurry and unfocused due to texture filtering.
Thursday, July 10, 2008
The Future of WED
WED 1.0 has gone RC. The on
ly change from beta 5 is that I have the latest manual changes from Tom (including Cormac's illustrations of taxiway signs). Both of them did some great work - the WED manual is a lot nicer than the docs I have done myself for the other tools.
A while ago I posted a tools update...has anything changed? Not a whole lot.
A DSF is a compiled result of a very complex set of processes. The vertex that you adjust in the mesh to fix a mountain top may not exist in the DSF if another user submits an updated airport layout, even if they are fifty kilometers apart. (All parts of the DSF affect each other through the adaptive irregular mesh.) So I don't want to take user submitted corrections to the final DSF because those changes would be lost at the next render.
We've been down this road before - when the V6 global scenery was done, the plan was: now we'll take user-edited ENVs. The problem was that this plan assumed that we'd never re-render the ENVs again, which proved totally wrong - we re-rendered them at the end of the v6 run with improved algorithms.
What we need to do is identify what components of the source data are reasonable candidates for user editing, and set up processes (similar to Robin's collection of apt.dat data) to gather data and share it.
(We are looking at OSM - it is still under investigation!)
ly change from beta 5 is that I have the latest manual changes from Tom (including Cormac's illustrations of taxiway signs). Both of them did some great work - the WED manual is a lot nicer than the docs I have done myself for the other tools.A while ago I posted a tools update...has anything changed? Not a whole lot.
- WED is definitely the future of the "heavy" scenery tools (ones with extensive UI) - it has a lot of infrastructure for features like multiple undo, multiple selection, hiearchial editing, etc.
- The next big feature for WED will be some kind of overlay editing. I don't think this will happen very soon though - my todo list is pretty out of control.
- In the long term, I think WED may provide a visual way to do MeshTool-like operations. In other words, you'll be able to build a base mesh in WED by specifying polygons for airports, applying orthophotos, and importing one big DEM per tile.
- I don't think that WED will ever be a DSF editor - that is, you won't be able to open an existing DSF, move a single node, and re-save it. This just isn't high on my priority list.
A DSF is a compiled result of a very complex set of processes. The vertex that you adjust in the mesh to fix a mountain top may not exist in the DSF if another user submits an updated airport layout, even if they are fifty kilometers apart. (All parts of the DSF affect each other through the adaptive irregular mesh.) So I don't want to take user submitted corrections to the final DSF because those changes would be lost at the next render.
We've been down this road before - when the V6 global scenery was done, the plan was: now we'll take user-edited ENVs. The problem was that this plan assumed that we'd never re-render the ENVs again, which proved totally wrong - we re-rendered them at the end of the v6 run with improved algorithms.
What we need to do is identify what components of the source data are reasonable candidates for user editing, and set up processes (similar to Robin's collection of apt.dat data) to gather data and share it.
(We are looking at OSM - it is still under investigation!)
Just What Is a Slider?
I've blogged in the past about the difference between key frames to animate changes to a dataref and key frames to animate an event. Simply put, key frames let you relate the position of your mesh to a dataref value, but they do not let you start a sequence of animations based on a change to a dataref.
In other words, you can make an aniamtion that puts the gear handle up when the gear dataref is 0 and down when the dataref is 1. If that dataref were ever 0.5 you' d see something in the middle. (This dataref will always be 0 or 1 though.)
But you can't use animation to make the handle slowly move from up to down over a few seconds when the dataref instantly jumps from 0 to 1. If the dataref moves instantly, so does your animation. There can be no time delay between the dataref and animation.
To get this "triggered sequence" effect, you need something else to detect the event and create a dataref that moves slowly. That something could be a plugin; X-Plane also provides a built-in mechanism: the "misc. slider".
In an attempt to put the information somewhere useful, I have described the feature on the X-Plane wiki.
No Spare Datarefs (Usually)
In that last blog post I went on a rant about "spare" datarefs. Basically people would ask us to make some datarefs that did absolutely nothing, simply to wire parts of their panels and animations together.
I relented on my usual hard-line position on this (and a few other) datarefs for two reasons:
Someone in the X-Plane community can write a (really simple) open-source plugin whose sole purpose is to create trivial datarefs for aircraft use from a text configuration file, letting aircraft authors make as many datarefs as they need. (It is possible that this plugin exists and I just don't know about it.)
Aircraft authors: to make sure that datarefs from differing add-ons do not have name conflicts, Sandy and I established a registry of plugin dataref prefixes. If you are going to add custom datarefs to your airplane, be sure to register a prefix, and only use that prefix. Do not start your datarefs with sim/XXX - that prefix is for X-Plane's use only.
(Side note: the X-Plane SDK website has been under some kind of automated attack for the last week or so with lots of bogus user sign-ups. I haven't had time to address this properly; if you are not registered on the SDK Wiki, I may not be able to approve new sign-ups for a few more days.)
In other words, you can make an aniamtion that puts the gear handle up when the gear dataref is 0 and down when the dataref is 1. If that dataref were ever 0.5 you' d see something in the middle. (This dataref will always be 0 or 1 though.)
But you can't use animation to make the handle slowly move from up to down over a few seconds when the dataref instantly jumps from 0 to 1. If the dataref moves instantly, so does your animation. There can be no time delay between the dataref and animation.
To get this "triggered sequence" effect, you need something else to detect the event and create a dataref that moves slowly. That something could be a plugin; X-Plane also provides a built-in mechanism: the "misc. slider".
In an attempt to put the information somewhere useful, I have described the feature on the X-Plane wiki.
No Spare Datarefs (Usually)
In that last blog post I went on a rant about "spare" datarefs. Basically people would ask us to make some datarefs that did absolutely nothing, simply to wire parts of their panels and animations together.
And while I'm going off on a tangent about authoring and datarefs, we will not be adding any "spare" datarefs.Well, the slider_on and slider_ratio datarefs are sort of "spare" datarefs - they're 24 slots that have no default function.
I relented on my usual hard-line position on this (and a few other) datarefs for two reasons:
- These datarefs are tied to a real (albeit really simple) bit of simulation code - that is, the code that moves the sliders over time. We had to add these datarefs to make the slider mechanism accessible.
- Because sliders are an airplane feature, there is a logical scope: the person who writes the airplane gets to decide what the sliders do, and no one else gets to muck with them. (Since the cycle time is an aircraft file attribute, no one else could sanely hope to utilize them.)
Someone in the X-Plane community can write a (really simple) open-source plugin whose sole purpose is to create trivial datarefs for aircraft use from a text configuration file, letting aircraft authors make as many datarefs as they need. (It is possible that this plugin exists and I just don't know about it.)
Aircraft authors: to make sure that datarefs from differing add-ons do not have name conflicts, Sandy and I established a registry of plugin dataref prefixes. If you are going to add custom datarefs to your airplane, be sure to register a prefix, and only use that prefix. Do not start your datarefs with sim/XXX - that prefix is for X-Plane's use only.
(Side note: the X-Plane SDK website has been under some kind of automated attack for the last week or so with lots of bogus user sign-ups. I haven't had time to address this properly; if you are not registered on the SDK Wiki, I may not be able to approve new sign-ups for a few more days.)
Wednesday, July 09, 2008
What Is a File Format
I was reading this blog post on the Microsoft Office binary file formats...to quote:
But then Joel discusses why the formats are the way they are...tell me some of these don't apply to DSF:
The File Format as API vs. Data Container
File formats can serve two fundamentally different purposes:
X-Plane Owns Its Own File Formats
One of the decisions I made early on in my work with the X-Plane scenery system was to not try to use open standards for most parts of the scenery system. (OBJ8, DSF, these are X-Plane specific inventions.) Does this make me a closed-source fascist? Perhaps, but the intent was not to lock people out of X-Plane.
Instead the goal was to separate the problems of interchange and data storage in the places where the two would come into conflict. What became immediately clear was that we couldn't meet all of our goals for performance, compatibility, and interoperability for a scenery file format with one giant scheme.
So instead we have DSF, whose goal is to be a cryptic but fast final step in the scenery pipeline, and we have DSF2Text and MeshTool, which take simpler text file formats, meant for interchange, and create DSFs.
The intended user of a MeshTool or DSF2Text text file is another program that is outputting its data format. Thus by forming a number of small links in a chain, no one file format is meant to be the "super-format" that has to do everything for everyone.
A normal programmer would conclude that Office’s binary file formats:Hrm - deliberately obfuscated, a product of a demented borg mind, created by an insanely bad programmer, and impossible to read or write correctly...he must be talking about DSF!
- are deliberately obfuscated
- are the product of a demented Borg mind
- were created by insanely bad programmers
- and are impossible to read or create correctly.
But then Joel discusses why the formats are the way they are...tell me some of these don't apply to DSF:
- They were designed to be fast on very old computers. (Sort of true for DSF - load time on low-end hardware was a DSF design goal.)
- They were designed to use libraries. (True - DSF is a lot less weird when you consider the layered structure, and relatively easy to work with via DSFLib.)
- They were not designed with interoperability in mind. (I'll get to this later.)
- They have to reflect all the complexity of the applications.
- They have to reflect the history of the applications.
The File Format as API vs. Data Container
File formats can serve two fundamentally different purposes:
- A file format can specify a data container, used to save information for use later. DSF fits this category: the purpose of the DSF is to hold scenery data in a compact and fast-to-read format for X-Plane. (The other goal of DSF is to be structured so that we don't have to change it all the time.)
- A file format can be an interchange standard between two programs, similar to a programming API. DSF2Text text files are like that.
X-Plane Owns Its Own File Formats
One of the decisions I made early on in my work with the X-Plane scenery system was to not try to use open standards for most parts of the scenery system. (OBJ8, DSF, these are X-Plane specific inventions.) Does this make me a closed-source fascist? Perhaps, but the intent was not to lock people out of X-Plane.
Instead the goal was to separate the problems of interchange and data storage in the places where the two would come into conflict. What became immediately clear was that we couldn't meet all of our goals for performance, compatibility, and interoperability for a scenery file format with one giant scheme.
So instead we have DSF, whose goal is to be a cryptic but fast final step in the scenery pipeline, and we have DSF2Text and MeshTool, which take simpler text file formats, meant for interchange, and create DSFs.
The intended user of a MeshTool or DSF2Text text file is another program that is outputting its data format. Thus by forming a number of small links in a chain, no one file format is meant to be the "super-format" that has to do everything for everyone.
Friday, July 04, 2008
Third Party Add-on Compatibility in 920
I am in Boston visiting family for the 4th of July, so there are a few compatibility bugs that I have seen but probably won't be able to fix for a few days. Austin is still cranking out betas. So if new betas come out and these are not fixed, don't panic - they are definitely on my todo list.
- Sceneries converted via FS2XPlane crash on load. It looks like this is due to a bug in the new threaded texture loader - I think you can work around this by turning off "load scenery in the background" in the rendering settings (but then loads will be slow). I have found the area where the bug occurs but haven't isolated it.
- A user submitted a plane to us that crashes Plane-Maker on open - the panel code gets confused. I haven't isolated it yet. If you have planes you've made, save the pre-920 versions!
- I think Benedikt's x737 plugin should start working again in beta 2.
Tuesday, July 01, 2008
902 Is Dead (Long Live 920) - Languages
We're about to do something strange: any time now, Austin should announce 920 beta 1.
But what happened to the 902 beta program?
The answer is simple: it turns out we don't need it.
Both 901 and 902 were tiny releases whose sole purpose was to add new languages to X-Plane. (901 added French/German, 902 Russian). The idea was to get our foreign distributors a localized version of the sim faster than we could do the next major patch.
Well, 920 is here, and the Russian port is still being debugged, so 920 will contain Russian and also a huge host of new features. There will never be a 902 final; simply update your beta to 920 when it's out - any tiny changes in 902 are baked into 920.
Don't Panic
As always, my advice for beta 1 is: don't panic. Remember that we are a small company, and thus we have a small number of computers between all of our employees. The first few betas usually involve one or two serious bugs (the sim explodes and sets your computer on fire, etc.) that only happen on the one hardware configuration none of us have.
If you are a user, consider waiting for a later beta, or upgrade a copy of the sim.
If you create third party add-ons, please get a copy of the beta early and start filing bugs! It is much easier for us to fix compatibility bugs when we get the reports early.
Multi-Lingual
A number of users have offered to help translate X-Plane. With 920 we finally have the "new" localization infrastructure in place; I will finally be able to get users started on this process.
But what happened to the 902 beta program?
The answer is simple: it turns out we don't need it.
Both 901 and 902 were tiny releases whose sole purpose was to add new languages to X-Plane. (901 added French/German, 902 Russian). The idea was to get our foreign distributors a localized version of the sim faster than we could do the next major patch.
Well, 920 is here, and the Russian port is still being debugged, so 920 will contain Russian and also a huge host of new features. There will never be a 902 final; simply update your beta to 920 when it's out - any tiny changes in 902 are baked into 920.
Don't Panic
As always, my advice for beta 1 is: don't panic. Remember that we are a small company, and thus we have a small number of computers between all of our employees. The first few betas usually involve one or two serious bugs (the sim explodes and sets your computer on fire, etc.) that only happen on the one hardware configuration none of us have.
If you are a user, consider waiting for a later beta, or upgrade a copy of the sim.
If you create third party add-ons, please get a copy of the beta early and start filing bugs! It is much easier for us to fix compatibility bugs when we get the reports early.
Multi-Lingual
A number of users have offered to help translate X-Plane. With 920 we finally have the "new" localization infrastructure in place; I will finally be able to get users started on this process.
Subscribe to:
Posts (Atom)