Discussion:
[Flightgear-devel] 2017.2 FGAddon model versioning
Richard Harrison
8 years ago
Permalink
With 2017.2 will models in FGAddon be placed into a different download
location (or earlier models left at pre 2017.2 version)

I ask because the HUD change in particular isn't compatible with pre
2017.2 and results in a large white area (on the F-15) where the HUD
should be.

What I'd expect (after 2017.2) is that 2017.1 would still work if I
update FGAddon models to be compatible with 2017.2. Is this the case?
James Turner
8 years ago
Permalink
Post by Richard Harrison
With 2017.2 will models in FGAddon be placed into a different download
location (or earlier models left at pre 2017.2 version)
I ask because the HUD change in particular isn't compatible with pre
2017.2 and results in a large white area (on the F-15) where the HUD
should be.
What I'd expect (after 2017.2) is that 2017.1 would still work if I
update FGAddon models to be compatible with 2017.2. Is this the case?
Uh, I think you are asking for per-version installs of aircraft packages? Unfortunately that is not how the system is currently setup, we assume aircraft are backwards compatible for the moment. Adding ‘minimum version of FG’ support would be possible of course but needs some new (simple) bits of C++ code and a catalog XML extension.

But, I also wonder if I’m understanding the problem correctly, maybe I have not.

Kind regards,
James
Richard Harrison
8 years ago
Permalink
Post by James Turner
Uh, I think you are asking for per-version installs of aircraft
packages?

Exactly what I'm asking about.
Post by James Turner
Unfortunately that is not how the system is currently setup, we
assume aircraft are backwards compatible for the moment.

Ah ok.

I understood previously that backwards compatibility wasn't that much
required in FGAddon; it was a case of moving forwards.

I can keep my models backwards compatible at a cost[1]. I think we
should consider making the launcher download from /2017.1 etc - and
populate only the currently folder from the FGAddon script.

----------------------
[1] Taking the example of the new HUD; this needs shader support that
will be missing from FGData on 2017.1; I can make the model work by
copying the shaders into F-15/Effects - but that seems to be wrong.
Without the shaders I get a completely white HUD.
James Turner
8 years ago
Permalink
As it seems that the launcher will already look in <Release>/ first it
seems that the simplest approach is to simply copy Aircraft/ to
<Release>/ on each release.
The launcher could be changed to have a "pre-release / beta models"
checkbox which would just use Aircraft/
I think there may be some confusion creeping in about what the package manager code (which the launcher is just a front-end to) does.

The package-manager tracks one or more independent catalogs. Each catalog lists:

- a set of FG versions it is compatible with (with some wildcard support, eg:)

2017.1.* (for all 2017.1 patch levels)
3.* (for all 3.x versions)

- a list of aircraft, which correspond to the URLs of zip files.

Intentionally, there is some decoupling here, the URLs are not relative, so several catalogs could share a zip file for whatever reason.

There is code to do a couple of things:

- if the catalog does not support the running version of FG (which happens after an upgrade, potentially), look for a new catalog at a URL based on the FG version.

- use an update URL in the catalog to try to obtain an updated version of it, which might describe where to find a corresponding catalog for a newer (or even, older) version of FG.

(Support for the latter needs some work, because we never had enough co-existing catalogs to exercise it)

The idea being we call always establish a *catalog* which claims to be compatible with the running version of FG. It’s then up to the catalog maintainer (or the scripts they run) to ensure that all the aircraft zips referenced by a given catalog are compatible with that version of FG.

Kind regards,
James
Richard Harrison
8 years ago
Permalink
I'd like to see if we can properly resolve the versioning problem prior
to the 2017.3 release.

There are some enhancements I've got to both the F-14 and F-15 that
aren't backwards compatible and I'm not going to release these until we
can be sure that folks using older versions (with the launcher for
download) will be able to carry on. I don't want to break old versions.

Unless I'm wrong, if you've downloaded 2017.1 and are using it with the
launcher; as soon as you install a new aircraft it will get the current
version[1] which won't work, which is a poor state of affairs. I know
everyone should be using the latest version; but they may not be and I'm
not going to break compatibility as I get enough support requests for
these two craft[2] without doing something that is going to make me look
like an idiot.

Don't know if I'd said it enough above; but I'm really not keen on
breaking old versions.

----------------
[1] because of what's on the http servers rather than the code.
[2] I'm not complaining about the amount of support requests - it's nice
that folks are flying my creations.
James Turner
8 years ago
Permalink
...
I agree, the pieces I have in progress already are:

<min-version> tag in aircraft -set.xml files

FlightGear checks these values on startup and shows a warning in the log and on-screen.

And on a local branch I have:

the launcher shows a message + alert icon on aircraft incompatible with the FG version, or tagged in certain ways

Still to be done:

make the catalog generation script in fgmeta/catalog extract min-version tags into the catalog
(this is easy)

All of this will solve the support issue, i hope. However the better thing would be to start versioning / tagging FGaddon and maintaining versioned zip-files for at least the past N releases. Again the catalog/zip creation scripts can do this already, but it does increase our server-side storage requirements considerably, and I believe we might be on uncertain territory them from Ibilbio’s side.

I agree with Richard that even if it’s not a live process, keeping the aircraft zips + catalogs for older releases around for say a year is desirable for a good user experience. (There’s actually no extra disk-space overhead than I can see to making it ‘live’, i.e re-checking the branch in FGaddon and regenerating the 1% of aircraft which do receive bug-fixes on the branch, but, again, I don’t think it matters either way if we do that step)

For this particular issue, the decision rests entirely with Curt (as the person with access to the Ibiblio server), so it comes down to, Curt, what is your opinion? In the past you’ve said you don’t think supporting older versions is worth it, personally I would say supporting them for a year would be a very good step.

Kind regards,
James
Curtis Olson
8 years ago
Permalink
We have entire aircraft trees for past versions of flightgear on our ftp
server, so I never meant to say supporting older versions of flightgear
isn't worth it in a universal sense.

There would need to be changes to the catalog building script, and then
we'd need to make some decisions and do some work with zip file
naming/storage conventions. Would the older aircraft versions live in
their own directory? Would everything live in a single directory, but
older aircraft have some different name? It would be nice if the catalog
building script would also support putting the files in the proper places
in a local ftp source tree so they can then be mirrored to the ftp server.

Best regards,

Curt.
...
Torsten Dreyer
8 years ago
Permalink
Dont know if it helps or if I am stating the obvious but we do create SVN
branches for FGAddon with every release:
https://sourceforge.net/p/flightgear/fgaddon/HEAD/tree/branches/
--
Torsten Dreyer
James Turner
8 years ago
Permalink
We have entire aircraft trees for past versions of flightgear on our ftp server, so I never meant to say supporting older versions of flightgear isn't worth it in a universal sense.
Ah ok, my apologies for that misunderstanding!

In which case
.
...
Well, my expectation would be to keep the trees separate, so we don’t need to change the script, we simply run it multiple times, with different config files:


fgaddon/branches/2017.1. -> catalogs/2017.1/***** (zips, thumbnails, catalog.xml)

fgaddon/branches/2016.2. -> catalogs/2017.2/***** (zips, thumbnails, catalog.xml)

.. and so on ..

Each of these produced by a separate invocation of the catalog script, running from a separate branch of FGaddon, and uploading to a separate file tree on the server.

I think this is ‘easy’, am I missing something? We just need to ensure each older catalog has the right forwards migration URL so when people upgrade their simulator, the catalog knows how to find the correct newer version. And there is probably a little bit of code tweaking / debugging needed, since we never tested that logic much so far!

Kind regards,
James

w***@gmail.com
8 years ago
Permalink
I think we will need to keep releases for a few years otherwise it
doesn't really achieve the object of allowing people time to upgrade
without getting potentially weird aircraft effects.
I've seen people still using FG-3.2 because their older hardware can't run
newer versions... make that lots of years!
yeah, but they're not trying to run the latest craft, either... i don't know of
any currently maintained craft that specifically detect and support anything
older than 3.4...
--
NOTE: No off-list assistance is given without prior approval.
*Please keep mailing list traffic on the list unless*
*a signed and pre-paid contract is in effect with us.*
Ludovic Brenta
8 years ago
Permalink
I don't recall if stable-branch vs. head for aircraft packaging was
specifically discussed. On the plus side, tracking head puts new
aircraft
improvements into the hands of users right away. For every one time
it
breaks an aircraft for some people on older versions, there are
probably 99
times it puts interesting or important improvements into the hands of
users
right away.
We could get the best of both worlds by changing our processes:

* the built-in launcher always looks in the branch of FGAddon
corresponding
to its release; nightly builds look in trunk;
* Aircraft developers actively develop in the latest stable release
branch
until they break compatibility with that branch.

This imposes additional work for aircraft developers: they would need
to
merge their development changes to all branches more recent than the
one
they develop in.

The Lockheed1049h is compatible with all releases of FlightGear that I
know
of, so by that rationale I should be making improvements in the oldest
stable branch, which is release-3.4.0, and merging my changes in all
subsequent branches. If I want to switch to multiplayer protocol
version 2,
I simply stop developing in that branch and start developing in
release-2017.2, and merge in all subsequent branches. Yes, that's a
lot of
work but scripts can automate the merging.

Aircraft developers outside of FGAddon who want to support multiple
versions
of FlightGear actually do variations of that already, just not
systematically
across 600+ aircraft :)
--
Ludovic Brenta.
Thorsten Renk
8 years ago
Permalink
On the plus side, tracking head puts new aircraft
improvements into the hands of users right away. For every one time it
breaks an aircraft for some people on older versions, there are probably
99 times it puts interesting or important improvements into the hands of
users right away.
... which the user otherwise would get no more than 3 month later anyway.

I've heard quite some buzz about doing day to day development on FGAddon
(and I'm so glad now I have not done this!) - to the degree that people
actually do it, it also ships any temporary breakage to the user right
away, and it ships inconsistent documentation and actual behavior to the
user right away.

Anyone who wants to track daily development of anything can do so by
tracking the repositories - but defaulting unwary users of a stable
release to a development repository seems really not a good idea to me -
especially with such a short release cycle the pain to gain ratio seems
large.


* Thorsten
Edward d'Auvergne
8 years ago
Permalink
...
Of course, the generation on the fly idea was just a suggestion for
minimising the storage requirements. This is actually what the
SourceForge infrastructure provides - the FGAddon svn web interface
has an auto-generated zip file available via the 'Download Snapshot'
link. However I believe that FGAddon is so big that this feature
fails on their server infrastructure (possibly a timeout). Maybe we
should talk to them and see if it could be fixed? Then you could use
URLs such as the following to auto-generate the download:

https://sourceforge.net/p/flightgear/fgaddon/HEAD/tarball?path=/branches/release-2016.1/Aircraft/F-15

As an example of this SF SVN web-interface feature working, here is a
link from one of my own projects:

https://sourceforge.net/p/nmr-relax/code-svn-mirror/HEAD/tarball?path=/0.1

Despite the 'tarball' in the URL, this generates a zip file. If you
follow that URL you'll see that the zip file is now cached - the first
time I followed that link it took 2-3 min to generate the zip file.
Another storage minimising idea is that because many of the ~500
aircraft do not change between versions, the old version zip file
could be a symlink to the current version.

As for the FGAddon SVN repository organisation with a /maintenance
directory, that idea is independent of the way the old version zip
files are created or stored. A scratch development area within
FGAddon where extensive branching is possible would be of benefit to a
number of aircraft developers who currently use the git-svn bridge
just to have multiple development branches. You could have an
unlimited number of branches within the /maintenance/Aircraft/F-15/
directory.

Regards,

Edward
Thorsten Renk
8 years ago
Permalink
Post by James Turner
Uh, I think you are asking for per-version installs of aircraft
packages? Unfortunately that is not how the system is currently setup,
we assume aircraft are backwards compatible for the moment.
Huh?

We know for a fact that they're not backward compatible - whenever we
introduce a new bit of technology, be it autopilot tags, effects or Nasal
functions in the core, aircraft using that functionality won't work with a
previous FG version which doesn't provide it. As in Richard's case with
the HUD.

The rule has always been that I can run an 2016.1 aircraft with a 2017.2
binary, but generally not a 2017.2 aircraft with a 2016.1 binary (might
still work dependent on aircraft).

Making this work aircraft-side is pretty ugly if you want to support a few
versions back and still have newest features.

So... then why do we have FGAddon branches for the releases if we don't
use them?

https://sourceforge.net/p/flightgear/fgaddon/HEAD/tree/branches/

* Thorsten
Thorsten Renk
8 years ago
Permalink
This post might be inappropriate. Click to display it.
chris
8 years ago
Permalink
I too thought, that we're still with the aircraft download page on https://flightgear.org, which presents all aircraft coupled to the corresponding version of FG. I haven't been looking there for a while, since I use plain svn for fgaddon. It seems, this has been dropped in favour of using the QTlauncher (which generally is fine with me) and as an alternative points to ftp:next.

I'd like to suggest, that maybe we can find some possibility to pull the versions of aircraft according to it's release branch. I think it would be good to stick to the mantra of keeping the aircraft only forward-compatible, as Thorsten illustrated.

FG has always been patient with slow aircraft developers, to pull along with latest technologies. The other way around would break most existing aircraft in the long run.

My 2 pennies. :o)

Cheers
chris
--
https://musicchris.de

GnuPG Fingerprint: FCFF A94E E987 33C0 F9A0 3FB3 D5FC DF84 9FB3 AC75
GnuPG Info: https://musicchris.de/index.php?page=blog&index=13
Wayne Bragg
8 years ago
Permalink
I've heard quite some buzz about doing day to day development on > FGAddon (and I'm so glad now I have not done this!) - to the degree
that people actually do it, it also ships any temporary breakage to >
the user right away, and it ships inconsistent documentation and >
actual behavior to the user right away.

My system for updating fgaddon repo aircraft always starts locally. I do
the work, test it, and after I am happy with it I push to fgaddon. For
the most part I have had little to no issues doing this. I remain
conscious of where the last stable release of FG is at so as not to push
a new feature that is not available yet.
So in essence, every commit I make to fgaddon is considered release ready.

What I like about this is I can introduce the latest upgrades available
in the current release if not yet implemented without delay as soon as I
implement them. I can fix any bug that pops up and know they are
available immediately.

But I do have to pay close attention.

However with Thorsten's approach, when working on the Shuttle, I don't
have to worry so much about introducing a bug, I know it is only going
to affect the immediate development branch.
From his point of view and the point of development the Shuttle is at,
I think it is a necessity to do it in release stages.

Listening to Thorsten's concerns does give me pause though. If I was
working on a critical system for some company, I would absolutely not do
it directly and would make sure it was well reviewed prior to
introducing it. But in the case of the aircraft I am responsible for,
guess what, I am the only one that reviews it and tests it unless I
introduce it to fgaddon where it gets tested by anyone that uses the sim
and the aircraft from that point forward (or from the point of
"update"). My worst case is a bug report on the forum and I fix it and
everyone is happy.

Bottom line is I see a use case for both approaches.

I wouldn't care if I had a way to update the "stable" version at any
time. If not, even though we're only talking a few months, I would not
be happy about not being able to change or add a feature in the stable
version until the next release.

One more piece to this to give a broader view. On the c172p, during its
most intensive development cycle, when we were submitting 1-10 PR a
week, our system of review before those PR's were allowed to be merged
kept the "main" branch "release ready" pretty much at all times anyway.
I would have no real fear about pointing to the development "main"
branch on that project.
I wouldn't even consider doing this with the Shuttle with the systems
and checks we have in place for it.
So it partially depends on the developers involved and the approaches
and systems they implement.

Wayne
Loading...