Increase speed in data-access using opensky.history()

[yweidmann]

Dear List

I am using the traffic-library to access the OpenSky Impala shell. I was placing this question in the forum of OpenSky-Network, but I have the impression, this will be the better place for it. Hope this is OK.

To retrieve flights arriving at a given airport during an entire day, I query the history of OpenSky with the given values:

• startTime = 2019-02-01 00:01
• endTime = 2019-02-01 23:59
• callsign = <value from previous query, see remarks below>
• icao24 = <value from previous query, see remarks below>
• bounds = <geographic box around the airport, about 6 by 3 degrees>

opensky.history(
    startTime,
    endTime,
    bounds = (5.0, 46.0, 11.0, 49.0),
    callsign = callsign,
    icao24 = icao24,
    count = False,
    return_flight = True)

Problem:

• The call opensky.history() can take several minutes.

Question:

• Is there an idea to speed up the query / call of opensky.history()?

Looking forward to any ideas and suggestions.

Best regards and many thanks in advance,
Yvo

Remarks:
As a previous query, I retrieve the entire list of flights during an entire day for the given arrival-airport.

airport = <airport-code>

opensky.flightlist(
    start = startTime,
    stop = endTime,
    arrival_airport = airport)

[xoolive]

Hello @yweidmann, I believe we are only subject to limitations on the Impala shell hosted by OpenSky here, so we just have to cope with it. I heard lines may be moving a bit soon though.

[yweidmann]

Perfect. Thank you very much for the fast answer.

[yweidmann]

Hello @xoolive, may I ask you another question regarding the limitations of the Impala shell:

Is it possible to access the shell by using the traffic-library in a multithreaded approach?

I tried a first simple approach, but I had to realize that the library uses cached data in the following directory:

• \traffic\traffic\Cache\opensky

In case of a multithreaded approach, the threads tried to access the same files, causing an exception.

The option cached = False did not solve the problem:

• flight = opensky.history(..., cached = False)
• flightList = opensky.flightlist(..., cached = False)

Question:

• Is it possible to run opensky.history() and opensky.flightlist() multithreaded?
• And if yes, would it be improving the download? Or would the shell not support it?

Looking forward to any suggestions.

Cheers,
Yvo

[xoolive]

Multithreading or asynchronous programming would be efficient when we have a lot of I/O, like many small requests: only then I/O overhead can be compensated by parallelism.

However, this is not the case with Impala queries which carry crazy amounts of data. That's why traffic chunks them so that you don't waste so much time when the connection breaks or stalls.

Doing multithreading with multiple open connections would actually:

• cut some insignific I/O overhead, which is peanuts compared to the data you query/bandwidth you are granted. In the end, the multithreading overhead would probably be slightly detrimental to your performance;
• see your account + IP banned from OpenSky.

Understand the bottleneck here is the amount of data you query.
Bandwidth is fixed. The capacity of the server too.

[yweidmann]

Hi @xoolive

Thank you very much for the fast replay and detailed description. I highly appreciate.

Cheers,
Yvo