Discussion:
[Flightgear-devel] Where to download apt.dat?
Torsten Dreyer
2017-07-10 09:40:05 UTC
Permalink
Hi,

as some of you are probably aware of, I am currently setting up a Jenkins
pipeline to automatically and regularly bake a new world scenery.

Does anybody know a way to pull the latest apt.dat from the x-plane gateway
with curl or some other automated method?

This page https://gateway.x-plane.com/airports/page allows downloading
single airports and the REST API returns JSON.

Torsten
--
Torsten Dreyer
w***@worldwideweb2.nl
2017-07-10 09:52:21 UTC
Permalink
He Torsten

I bereave you need to look in this project:
https://github.com/accek/fg-nav-fixups

Jean-Paul Anceaux (www2)
Post by Torsten Dreyer
Hi,
as some of you are probably aware of, I am currently setting up a Jenkins
pipeline to automatically and regularly bake a new world scenery.
Does anybody know a way to pull the latest apt.dat from the x-plane gateway
with curl or some other automated method?
This page https://gateway.x-plane.com/airports/page allows downloading
single airports and the REST API returns JSON.
Torsten
--
Torsten Dreyer
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org!
http://sdm.link/slashdot_______________________________________________
Flightgear-devel mailing list
https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Torsten Dreyer
2017-07-10 11:39:47 UTC
Permalink
Post by w***@worldwideweb2.nl
https://github.com/accek/fg-nav-fixups
Seriously? Compile apt.dat from JSON? That does not sound right.
--
Torsten Dreyer
Michael
2017-07-10 12:16:15 UTC
Permalink
I used to monitor that data to keep my project up to date as it was packed
in one huge .dat file with all airports and up for download on the X-plane
portal. The last version I ever saw available was just over a year ago,
that time only accessible via a "NOTAM" on an X-plane dev page some time in
the spring.

I also noted the main difference was a ~50% increase in size (!), with main
added content being:
- "linear features" = mostly TWY centre lines
- some ground route networks

That added ~40 MB of text data to enhance a portion only of the available
airports. I started to wonder how they would sustain the
whole-world-in-one-file strategy... Then I saw the airport gateway search
interface improved, and never heard of a unique apt.dat again, and lost
hold of the link to it. I therefore figured this download was (probably
rightly) discontinued.

Compiling whole world apt data may now be at the expense of checking for
them each individually, unless we can gain privileged access to the X-plane
DB?
Post by Torsten Dreyer
Post by w***@worldwideweb2.nl
https://github.com/accek/fg-nav-fixups
Seriously? Compile apt.dat from JSON? That does not sound right.
--
Torsten Dreyer
------------------------------------------------------------
------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Flightgear-devel mailing list
https://lists.sourceforge.net/lists/listinfo/flightgear-devel
www2
2017-07-10 12:54:26 UTC
Permalink
Torsten the json file contain a binary blob (zip file) that contain the
apt.dat file for the airport.
For the file we have full read access of the database for an relese we
can do the foiling:
1. download the list with the release from
https://gateway.x-plane.com/apiv1/releases and select the latest release
2. select the latest release and download the list with the scenery
packs https://gateway.x-plane.com/apiv1/release/11.00
3. download the scenery pack from
http://gateway.x-plane.com/apiv1/scenery/{scenery_pack_id} and open the
zip file and get apt.dat for the airport.

Jean-Paul Anceaux (www2)
Post by Michael
I used to monitor that data to keep my project up to date as it was
packed in one huge .dat file with all airports and up for download on
the X-plane portal. The last version I ever saw available was just over
a year ago, that time only accessible via a "NOTAM" on an X-plane dev
page some time in the spring.
I also noted the main difference was a ~50% increase in size (!), with
- "linear features" = mostly TWY centre lines
- some ground route networks
That added ~40 MB of text data to enhance a portion only of the
available airports. I started to wonder how they would sustain the
whole-world-in-one-file strategy... Then I saw the airport gateway
search interface improved, and never heard of a unique apt.dat again,
and lost hold of the link to it. I therefore figured this download was
(probably rightly) discontinued.
Compiling whole world apt data may now be at the expense of checking for
them each individually, unless we can gain privileged access to the
X-plane DB?
https://github.com/accek/fg-nav-fixups
<https://github.com/accek/fg-nav-fixups>
Seriously? Compile apt.dat from JSON? That does not sound right.
--
Torsten Dreyer
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Flightgear-devel mailing list
https://lists.sourceforge.net/lists/listinfo/flightgear-devel
<https://lists.sourceforge.net/lists/listinfo/flightgear-devel>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Flightgear-devel mailing list
https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Torsten Dreyer
2017-07-10 13:34:44 UTC
Permalink
Sigh.
I am beginning to understand the fun factor of creating world-scenery.
--
Torsten Dreyer
Torsten Dreyer
2017-07-10 15:13:52 UTC
Permalink
Thanks Jean-Paul
with that algorithm you layed out, it would be
a) 1 http GET request to get the current release
b) 1 http GET request to get the list of released airports
c) 35685 http GET requests to pull all available airport definitions.

I would not mind a) and b) but I am getting slight nervous about c)
Post by Torsten Dreyer
Sigh.
I am beginning to understand the fun factor of creating world-scenery.
--
Torsten Dreyer
--
Torsten Dreyer
Curtis Olson
2017-07-10 15:22:42 UTC
Permalink
Hi Torsten,

I might be worth searching our devel list for past messages from Ben Supnik
(sp?). He is an x-plane developer that has been friendly to our apt.dat
usage. Life has gotten more complicated as the x-plane folks have added
more and more details and fields to the apt.dat file and consider some of
the categories proprietary to x-plane. I think there would be ways forward
to make this more efficient and not beat up on their server to the point
that they'd block our IP address. It may involve a little discussion with
Ben though to find a good way forward?

Thanks,

Curt.
Post by Torsten Dreyer
Thanks Jean-Paul
with that algorithm you layed out, it would be
a) 1 http GET request to get the current release
b) 1 http GET request to get the list of released airports
c) 35685 http GET requests to pull all available airport definitions.
I would not mind a) and b) but I am getting slight nervous about c)
Post by Torsten Dreyer
Sigh.
I am beginning to understand the fun factor of creating world-scenery.
--
Torsten Dreyer
--
Torsten Dreyer
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Flightgear-devel mailing list
https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Torsten Dreyer
2017-07-10 15:37:26 UTC
Permalink
Thanks, Curt - I'll try to contact Ben.
Post by Curtis Olson
Hi Torsten,
I might be worth searching our devel list for past messages from Ben
Supnik (sp?). He is an x-plane developer that has been friendly to our
apt.dat usage. Life has gotten more complicated as the x-plane folks have
added more and more details and fields to the apt.dat file and consider
some of the categories proprietary to x-plane. I think there would be ways
forward to make this more efficient and not beat up on their server to the
point that they'd block our IP address. It may involve a little discussion
with Ben though to find a good way forward?
Thanks,
Curt.
Post by Torsten Dreyer
Thanks Jean-Paul
with that algorithm you layed out, it would be
a) 1 http GET request to get the current release
b) 1 http GET request to get the list of released airports
c) 35685 http GET requests to pull all available airport definitions.
I would not mind a) and b) but I am getting slight nervous about c)
Post by Torsten Dreyer
Sigh.
I am beginning to understand the fun factor of creating world-scenery.
--
Torsten Dreyer
--
Torsten Dreyer
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Flightgear-devel mailing list
https://lists.sourceforge.net/lists/listinfo/flightgear-devel
--
Torsten Dreyer
Florent Rougon
2017-07-10 18:17:06 UTC
Permalink
Hi,
Post by Torsten Dreyer
Thanks, Curt - I'll try to contact Ben.
Yes, Ben explained here some time ago (~1 year) that preparing the
monolithic apt.dat took X-Plane developers much time, so they didn't do
it anymore (at least, not often). IIRC, he also said they would have no
problem with us reading the whole db from their gateway (1 download per
airport) if we were well-behaving (my wording), with only one person
doing it for the FG project.

Also, Jean-Paul pointed you to Szymon Acedański's work (for apt.dat:
<https://github.com/accek/fg-nav-fixups/tree/master/xplane-apts>) that
seems to cache airports:

,----
| xplane-get-apts
|
| This script downloads all the recommended sceneries from the X-Plane Gateway
| and concatenates them into a single apt.dat.
|
| The downloaded airports are cached and the process may be stopped and resumed
| at any time.
|
| To create an updated airports file, delete out/json.meta and run the script
| -- only the updated airports will be downloaded from the gateway.
`----

I haven't looked at this Python code(*), just tested FG with the
resulting apt.dat file at the time. Szymon did good work in the nav.dat
and fix.dat areas (see FG commits from February), so if he's still
available and willing, it could be good to work with him.

(*) And don't know how it differs from the script Alessandro pointed to.

One piece of code Szymon wrote that I didn't have time to review and
use, is this script:

https://github.com/accek/fg-nav-fixups/blob/master/ils-fixup.py

The goal was to make sure it is safe to get rid of the *.ils.xml files
in TerraScenery, which at runtime override data from nav.dat files and
can thus be confusing (this was discovered while examining an LDA
problem at VTSP [Phuket International Airport], IIRC). I'm awfully sorry
I couldn't follow up on Szymon's work here. I never forgot it, but had
to order priorities hardly to avoid other work being wasted... So, if
someone wants to handle this little problem, he is more than welcome as
far as I'm concerned.

I'll follow up a little bit for the i18n stuff we've discussed with
James, but unfortunately can't engage on any other front until things
get better for me.

Regards
--
Florent
Torsten Dreyer
2017-07-10 19:54:41 UTC
Permalink
Hi all,

thanks for your comments. Looks like I have to assemble that file here
locally using one of the mentioned python scripts.
However, I'll first negotiate with Ben that he is ok with that. I want to
take their statement seriously:

<quote>
*Please note*: There is currently no metering on the Gateway API, but we
ask that you be considerate of our server load and avoid making unnecessary
requests. (For instance, if you start making frequent requests to get all
34,000 airports in the database, we’ll have to cut you off!)
</quote>

Regards
Post by Florent Rougon
Hi,
Post by Torsten Dreyer
Thanks, Curt - I'll try to contact Ben.
Yes, Ben explained here some time ago (~1 year) that preparing the
monolithic apt.dat took X-Plane developers much time, so they didn't do
it anymore (at least, not often). IIRC, he also said they would have no
problem with us reading the whole db from their gateway (1 download per
airport) if we were well-behaving (my wording), with only one person
doing it for the FG project.
<https://github.com/accek/fg-nav-fixups/tree/master/xplane-apts>) that
,----
| xplane-get-apts
|
| This script downloads all the recommended sceneries from the X-Plane Gateway
| and concatenates them into a single apt.dat.
|
| The downloaded airports are cached and the process may be stopped and resumed
| at any time.
|
| To create an updated airports file, delete out/json.meta and run the script
| -- only the updated airports will be downloaded from the gateway.
`----
I haven't looked at this Python code(*), just tested FG with the
resulting apt.dat file at the time. Szymon did good work in the nav.dat
and fix.dat areas (see FG commits from February), so if he's still
available and willing, it could be good to work with him.
(*) And don't know how it differs from the script Alessandro pointed to.
One piece of code Szymon wrote that I didn't have time to review and
https://github.com/accek/fg-nav-fixups/blob/master/ils-fixup.py
The goal was to make sure it is safe to get rid of the *.ils.xml files
in TerraScenery, which at runtime override data from nav.dat files and
can thus be confusing (this was discovered while examining an LDA
problem at VTSP [Phuket International Airport], IIRC). I'm awfully sorry
I couldn't follow up on Szymon's work here. I never forgot it, but had
to order priorities hardly to avoid other work being wasted... So, if
someone wants to handle this little problem, he is more than welcome as
far as I'm concerned.
I'll follow up a little bit for the i18n stuff we've discussed with
James, but unfortunately can't engage on any other front until things
get better for me.
Regards
--
Florent
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Flightgear-devel mailing list
https://lists.sourceforge.net/lists/listinfo/flightgear-devel
--
Torsten Dreyer
www2
2017-07-10 15:58:44 UTC
Permalink
We can make an gateway mirror that check ones a hour for new airports
and create a apt.dat after eatch x-plane release.

About the server so long we do in a serial sequential with one
connection i be leave that X-Plane have no problem what we do.

But i agree that is better ask than sory.

Jean-Paul Anceaux (www2)
Post by Curtis Olson
Hi Torsten,
I might be worth searching our devel list for past messages from Ben
Supnik (sp?). He is an x-plane developer that has been friendly to our
apt.dat usage. Life has gotten more complicated as the x-plane folks
have added more and more details and fields to the apt.dat file and
consider some of the categories proprietary to x-plane. I think there
would be ways forward to make this more efficient and not beat up on
their server to the point that they'd block our IP address. It may
involve a little discussion with Ben though to find a good way forward?
Thanks,
Curt.
Thanks Jean-Paul
with that algorithm you layed out, it would be
a) 1 http GET request to get the current release
b) 1 http GET request to get the list of released airports
c) 35685 http GET requests to pull all available airport definitions.
I would not mind a) and b) but I am getting slight nervous about c)
Sigh.
I am beginning to understand the fun factor of creating world-scenery.
--
Torsten Dreyer
--
Torsten Dreyer
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org!
http://sdm.link/slashdot_______________________________________________
Flightgear-devel mailing list
https://lists.sourceforge.net/lists/listinfo/flightgear-devel
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Flightgear-devel mailing list
https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Alessandro Menti
2017-07-10 17:53:04 UTC
Permalink
Hi,
this script can also get you a complete apt.dat file (and has caching,
so you won't need to download a new complete copy each time):
https://github.com/k-popov/xplane2fgfs
There is no way to download a complete apt.dat from the Web, as it is
downloaded automatically via the X-Plane installer; as Curt said, it
would be better to contact Ben Supnik and ask him to give you the URL.

Fixes and navigation files, on the contrary, can be downloaded from the
NOTAMs page of the Gateway:
https://gateway.x-plane.com/NOTAMs
(however, the latest update dates back to March 6th, and is in the old
X-Plane 10 format).

Cheers,
Alessandro Menti
Loading...